Method and apparatus for tracking of food intake and other behaviors and providing relevant feedback

ABSTRACT

A sensing device monitors and tracks food intake events and details. A processor, appropriately programmed, controls aspects of the sensing device to capture data, store data, analyze data and provide suitable feedback related to food intake. More generally, the methods might include detecting, identifying, analyzing, quantifying, tracking, processing and/or influencing, related to the intake of food, eating habits, eating patterns, and/or triggers for food intake events, eating habits, or eating patterns. Feedback might be targeted for influencing the intake of food, eating habits, or eating patterns, and/or triggers for those. The sensing device can also be used to track and provide feedback beyond food-related behaviors and more generally track behavior events, detect behavior event triggers and behavior event patterns and provide suitable feedback.

CROSS-REFERENCES TO PRIORITY AND RELATED APPLICATIONS

This application claims priority from and is a non-provisional of U.S.Provisional Patent Application No. 62/288,408 filed Jan. 28, 2016entitled “Method and Apparatus for Food Intake Tracking and Feedback”.The entire disclosures of applications recited above is herebyincorporated by reference, as if set forth in full in this document, forall purposes.

FIELD OF THE INVENTION

The present invention relates generally to electronic devices thatrelate to health technology and more particularly to methods andapparatus for using sensors for tracking a person's food intake, aprocessor for analyzing a food intake process and electronic circuitsfor providing feedback to the person. The methods and apparatus canextend beyond just food intake.

BACKGROUND

Diet-related health issues have become one of the top global publichealth issues. In the past couple of decades, there has been a dramaticsurge in obesity and other diet-related health issues. According to theCenter for Disease Control (CDC), in 2011-2012 69% of all Americanadults age 20 and over were overweight and more than one third ofAmerican adults were obese. Obesity can lead to many health issues suchas for example cardiovascular diseases, Type 2 diabetes, hypertension,cancers, respiratory problems, gallbladder disease and reproductivecomplications. While there may be multiple factors leading to orcontributing to obesity, one critical factor is a person's behavior asit relates to food intake.

Over the years, several attempts have been made to track food andnutrition intake. One common way for a person to track their food intakeis to maintain a written diary. There are several issues with thisapproach. First of all, the accuracy of human-entered information tendsto be limited. Secondly, maintaining a written diary is cumbersome andtime-consuming, causing many users to drop out after a short period oftime. Thirdly, there is no mechanism for real-time feedback. Fourthly,they do not provide any insights into important aspects of eatingbehavior, such as the pace of eating.

More recently, software, typically installed on or accessed from atablet, mobile phone, laptop or computer, can be used to facilitate thelogging and tracking of a person's food intake. Such softwareapplications typically utilize a database that contains nutrient andcaloric information for a large number of food items. Unfortunately,devices and software to facilitate food journaling are often timescumbersome to use and require a lot of human intervention, such asmanual data entry or look up. They are furthermore mostly focused onfood intake content and portion tracking and do not provide insight intoother aspects of eating behavior such as the number of bites or the paceof eating. They also lack the ability to provide real-time feedbackabout eating habits or behavior.

Devices and methods that attempt to reduce the burden of manual dataentry or data look-up exist and provide another approach to obtaininglog data about food consumption. As an example, tableware and utensilswith built-in sensors have been proposed to track food intake moreautomatically. For example, a plate with integrated sensors andcircuitry might automatically quantify and track the content of foodthat is placed on the plate. Similarly, integrated sensors in a drinkingvessel might identify, quantify and track the contents of liquid in thecup. In another example, an eating utensil includes sensors that countthe number of bites taken by a person using the eating utensil. Thesemethods might fall short in not being able to automatically identify andquantify the content of the food being consumed and also only apply to alimited set of meal scenarios and dining settings and are not wellsuited to properly cover the wide range of different meal scenarios anddining settings that a typical person may encounter during a day.

Being able to handle a wide variety of meal scenarios and settings isimportant for seamless and comprehensive food intake tracking. A methodbased on an eating utensil may not be able to properly track the intakeof drinks, snacks or finger foods and such methods may also interferewith a person's normal social behavior. For example, it might not besocially acceptable for a user to bring their own eating utensils to arestaurant or a dinner at a friend's house.

Devices and methods have been described that quantify and track foodintake based on analysis of images of food taken by a portable devicethat has imaging capabilities, such as an app that runs on a mobilephone or tablet that has a camera. Some devices might use spectroscopyto identify food items based on their molecular makeup. Such devices mayuse crowd sourcing and/or computer vision techniques, sometimescomplemented with other image processing techniques, to identify a fooditem, estimate its nutritional content and/or estimate its portion size.However, many of these devices and methods are fond lacking in usabilityand availability in certain social settings.

While today's spectroscopy technology has been sufficiently miniaturizedto be included in portable devices, devices based on spectroscopy dohave a number of significant issues. First of all, such devices requirea significant amount of human intervention and cannot be easily used ina discreet way. In order to produce an accurate spectrographmeasurement, the person eating is required to hold the spectrometer fora few seconds close to or in contact with each food item they desire toidentify. Since the light generated by such portable spectrometers canonly penetrate up to a few centimeters into the food, multiplemeasurements are required for food items that do not have a homogeneouscomposition and thus a portable spectrometer would not work well forsandwiches, layered cakes, mixed salads, etc. Such human intervention isintrusive to the dining experience and may not be acceptable in manydining settings.

Improved methods and apparatus for food intake monitoring and analysisare needed.

SUMMARY

A sensing device monitors and tracks food intake events and details. Aprocessor, appropriately programmed, controls aspects of the sensingdevice to capture data, store data, analyze data and provide suitablefeedback related to food intake. More generally, the methods mightinclude detecting, identifying, analyzing, quantifying, tracking,processing and/or influencing, related to the intake of food, eatinghabits, eating patterns, and/or triggers for food intake events, eatinghabits, or eating patterns. Feedback might be targeted for influencingthe intake of food, eating habits, or eating patterns, and/or triggersfor those. The sensing device can also be used to track and providefeedback beyond food-related behaviors and more generally track behaviorevents, detect behavior event triggers and behavior event patterns andprovide suitable feedback.

The following detailed description together with the accompanyingdrawings will provide a better understanding of the nature andadvantages of the present invention.

BRIEF DESCRIPTION OF THE DRAWINGS

Various embodiments in accordance with the present disclosure will bedescribed with reference to the drawings, in which:

FIG. 1 is an illustrative example of an environment in accordance withat least one embodiment.

FIG. 2 is an illustrative example of a block diagram in which variousembodiments can be implemented.

FIG. 3 is an illustrative example of an environment in accordance withat least one embodiment.

FIG. 4 is an illustrative example of an environment that includescommunication with at least one additional device over the internet inaccordance with at least one embodiment.

FIG. 5 is an illustrative example of an environment where a food intakemonitoring and tracking device communicates directly with a base stationor an access point in accordance with at least one embodiment.

FIG. 6 is an illustrative example of a high-level block diagram of amonitoring and tracking device in accordance with at least oneembodiment.

FIG. 7 is an illustrative example of a block diagram of a monitoring andtracking device in accordance with at least one embodiment.

FIG. 8 shows an example of a machine classification system in accordancewith at least one embodiment of the present disclosure.

FIG. 9 shows an example of a machine classification training subsystemin accordance with at least one embodiment of the present disclosure.

FIG. 10 shows an example of a machine classification detector subsystemin accordance with at least one embodiment of the present disclosure.

FIG. 11 shows an example of a machine classification training subsystemthat uses, among other data, non-temporal data.

FIG. 12 shows an example of a machine classification detector subsystemthat uses, among other data, non-temporal data.

FIG. 13 shows an example of a training subsystem for an unsupervisedclassification system in accordance with at least one embodiment of thepresent disclosure.

FIG. 14 shows an example of a detector subsystem for an unsupervisedclassification system in accordance with at least one embodiment of thepresent disclosure.

FIG. 15 shows an example of a classifier ensemble system.

FIG. 16 shows an example of a machine classification system thatincludes a cross-correlated analytics sub-system.

DETAILED DESCRIPTION

In the following description, various embodiments will be described. Forpurposes of explanation, specific configurations and details are setforth in order to provide a thorough understanding of the embodiments.However, it will also be apparent to one skilled in the art that theembodiments may be practiced without the specific details. Furthermore,well-known features may be omitted or simplified in order not to obscurethe embodiment being described.

Various examples are provided herein of devices that a person would useto monitor, track, analyze and provide feedback on food intake, theintake process and timing and other relevant aspects of a person'seating, drinking and other consumption for various ends, such asproviding diet information and feedback. The data related to food intakeprocess might include, timing of the eating process, pace of eating,time since last food intake event, what is eaten, estimates of thecontents of what is eaten, etc. While a lot of the examples describedherein are related to food intake events, the methods and devicesdescribed herein are also applicable to other behavior events such asbrushing teeth, smoking, biting nails, etc. Data can be obtained fromsome stationary device having sensors and electronics, some mobiledevice having sensors and electronics that is easily moved and carriedaround by a person, and/or from wearable devices having sensors andelectronics that a person attaches to their person or clothing, or ispart of the person's clothing. In general, herein such devices arereferred to as sensing devices. Herein, the person having such a deviceand who's consumption is being monitored is referred to as the user butit should be understood that the device might be used unchanged insituations where the person consuming, the person monitoring, and theperson evaluating feedback need not all be the same person. Herein, whatis consumed is referred to as food intake, but it should be clear thatthese devices can be used to more generally track consumption andconsumption patterns. A behavior tracking/feedback system as describedherein might comprise one or more wearable devices and might alsocomprise one or more additional devices that are not worn. Theseadditional devices might be carried by the wearer or kept nearby so thatthey can communicate with the wearable devices. The behaviortracking/feedback system might also comprise remote elements, such as aremote cloud computing element and/or remote storage for userinformation.

A wearable device might be worn at different locations on the wearer'sbody (i.e., the person monitoring their behavior) and the wearabledevice might be programmed or configured to account for thosedifferences, as well as differences from wearer to wearer. For example,a right-handed person may wear the device around his right wrist whereasa left-handed person may wear the device around his left wrist. Usersmay also have different preferences for orientation. For example, someusers may want the control buttons on one side, whereas other users mayprefer the control buttons on the opposite side. In one embodiment, theuser may manually enter the wrist preference and/or device orientation.

In another embodiment, the wrist preference and/or device orientationmay be determined by asking the user to perform one or more pre-definedgestures and monitoring the sensor data from the wearable devicecorresponding to the user performing the pre-defined gesture or set ofgestures. For example, the user may be asked to move his hand towardshis mouth. The change in accelerometer sensor readings across one ormore axes may then be used to determine the wrist and deviceorientation. In yet another example, the behavior tracking/feedbacksystem may process the sensor readings from the wearable device whilethe user is wearing the device for a certain duration of time.Optionally, the behavior tracking/feedback system may further combinethe sensor readings with other data or metadata about the wearer, toinfer the wrist and device orientation. For example, the behaviortracking/feedback system may monitor the user for one day and record theaccelerometer sensor readings across one or more of the axes.

Since the movement of the lower arm is constrained by the elbow andupper arm, some accelerometer readings will be more frequent than othersbased on the wrist and device orientation. The information of theaccelerometers can then be used to determine the wrist and/or deviceorientation. For example, the mean, minimum, maximum and/or standarddeviation of the accelerometer readings could be used to determine thewrist and/or device orientation.

In some embodiments, sensing devices can sense, without requiring userinteraction, the start/end of a food intake event, the pace of eating,the pace of drinking, the number of bites, the number of sips, theestimation of fluid intake, and/or estimation of portion sizing.Operating with less human intervention, no human intervention, or onlyintervention not apparent to others will allow the devices to scale wellwith different meal scenarios and different social situations. Sensingmight include capturing details of the food before it is consumed, aswell as user actions that are known to accompany eating, such asrepeated rotation of an upper arm or other hand-to-mouth motions.Sensors might include an accelerometer, a gyroscope, a camera, and othersensors.

Using the devices can provide a person with low friction-of-use todetect, quantify, track and provide feedback related to the person'sfood intake content as well as the person's food intake behavior. Suchmethods have the potential of preventing, treating and, in certaincases, even curing diet-related diseases. Such devices can improveefficacy, accuracy and compliance, and reduce the burden of usage and toimprove social acceptance. The devices can operate autonomously with no,or very minimal, human intervention, and do not interfere in an invasiveor otherwise significant negative way with a person's normal activitiesor social interactions or intrude on the person's privacy. The devicesare able to handle a wide range of meal scenarios and dining settings ina discreet and socially-acceptable manner, and are capable of estimatingand tracking food intake content and quantity as well as other aspectsof eating behavior. The devices can provide both real-time andnon-real-time feedback to the person about their eating behavior, habitsand patterns.

It is generally known and understood that certain eating behaviors canbe linked to, triggered by or otherwise be influenced by physical,mental or environmental conditions such as for example hunger, stress,sleep, addiction, illness, physical location, social pressure, andexercise. These characteristics can form inputs to the processingperformed by or for the devices.

The devices might be useful for a person concerned about their diet. Forexample, people with Type 1 diabetes are usually on an insulin therapywhere, based on their food intake and other factors, they administer theproper insulin dosage. While the cause of Type 1 diabetes may not bedirectly linked to a person's eating behavior, a person with Type 1diabetes needs to carefully track his or her food intake in order tomanage his or her insulin therapy. Such patients will also benefit fromeasier to use and more discreet methods for food intake tracking. Insome embodiments of the sensing devices, the sensing device is part of afeedback-driven automated insulin delivery therapy system. Such a systemmight include continuous monitoring of a patient's glucose levels, aprecision insulin delivery system, and the use of insulin that has afaster absorption rate, that would further benefit from information thatcan be extracted from automated and seamless food intake tracking, suchas the tracking of carbohydrates and sugar intake. The devices mightalso be useful for wellness programs and the like.

A food intake event generally relates to a situation, circumstance oraction whereby a person eats, drinks or otherwise takes into his or herbody an edible substance. Edible substances may include, but are notlimited to, solid foods, liquids, soups, drinks, snacks, medications,vitamins, drugs, herbal supplements, finger foods, prepared foods, rawfoods, meals, appetizers, main entrees, desserts, candy, breakfast,sports or energy drinks. Edible substances include, but are not limitedto, substances that may contain toxins, allergens, viruses, bacteria orother components that may be harmful to the person, or harmful to apopulation or a subset of a population. Herein, for readability, food isused as an example of an edible substance, but it should be understoodthat other edible substance might be used instead of food unlessotherwise indicated.

Eating habits and patterns generally relate to how people consume food.Eating habits and patterns may include, but are not limited to, the paceof eating or drinking, the size of bites, the amount of chewing prior toswallowing, the speed of chewing, the frequency of food intake events,the amount of food consumed during a food intake event, the position ofthe body during a food intake event, possible movements of the body orof specific body parts during the food intake event, the state of themind or body during a food intake event, and the utensils or otherdevices used to present, handle or consume the food. The pace of eatingor drinking might be reflected in the time between subsequent bites orsips.

Triggers generally relate to the reasons behind the occurrence of a foodintake event, behind the amount consumed and behind how it is consumed.Triggers for food intake events and for eating habits or patterns mayinclude, but are not limited to, hunger, stress, social pressure,fatigue, addiction, discomfort, medical need, physical location, socialcontext or circumstances, odors, memories or physical activity. Atrigger may coincide with the food intake event for which it is atrigger. Alternatively, a trigger may occur outside the food intakeevent window, and might occur prior to or after the food intake event ata time that may or may not be directly related to the time of the foodintake event.

In some embodiments of the sensing device or system, fewer than all ofthe features and functionality presented in this disclosure areimplemented. For example, some embodiments may focus solely on detectionand/or processing and tracking of the intake of food without intendingto steer the user to modify his or her food intake or without tracking,processing or steering eating habits or patterns.

In many examples herein, the setting is that an electronic device isprovided to a user, who wears the electronic device, alone or while itis in communication with a nearby support device that might or might notbe worn, such as a smartphone for performing operations that the wornelectronic device offloads. In such examples, there is a person wearingthe electronic device and that person is referred to as the “wearer” inthe examples and the system comprises a worn device and may includeother components that are not worn and are nearby and components thatare remote, preferably able to communicate with the worn device. Thus,the wearer wears the electronic device, the electronic device includessensors, which sense environment about the wearer. That sensing can beof ambient characteristics, body characteristics, movement and othersensed signals as described elsewhere herein.

In many examples, functionality of the electronic device might beimplemented by hardware circuitry, or by program instructions that areexecuted by a processor in the electronic device, or a combination.Where it is indicated that a processor does something, it may be thatthe processor does that thing as a consequence of executing instructionsread from an instruction memory wherein the instructions provide forperforming that thing. While other people might be involved, a commonexample here is where the wearer of the electronic device is using thatelectronic device to monitor their own actions, such as gestures,behavior events comprising a sequence of gestures, activities, starts ofactivities or behavior events, stops of activities or behavior events,etc. Where it is described that a processor performs a particularprocess, it may be that part of that process is done separate from theworn electronic device, in a distributed processing fashion. Thus, adescription of a process performed by a processor of the electronicdevice need not be limited to a processor within the worn electronicdevice, but perhaps a processor in a support device that is incommunication with the worn electronic device.

FIG. 1 shows a high level functional diagram of a dietary tracking andfeedback system in accordance with an embodiment of the presentinvention. A system for dietary tracking and feedback may in partinclude one or more of the following: a food intake event detectionsubsystem 101, one or more sensors 102, a tracking and processingsubsystem 103, a feedback subsystem 106, one or more data storage units104 and a learning subsystem that might perform non-real-time analysis.In some embodiments, elements shown in FIG. 1 are implemented inelectronic hardware, while in others some elements are implemented insoftware and executed by a processor. Some functions might sharehardware and processor/memory resources and some functions might bedistributed. Functionality might be fully implemented in a sensordevice, or functionality might be implemented across the sensor device,a processing system that the sensor device communicates with, such as asmartphone, and/or a server system that handles some functionalityremote from the sensor device. For example, a wearable sensor devicemight make measurements and communicate them to a mobile device, whichthen uploads them over the Internet to a server that further processesthe data. Data or other information may be stored in a suitable format,distributed over multiple locations or centrally stored, in the formrecorded, or after some level of processing. Data may be storedtemporarily or permanently.

A first component of the system illustrated in FIG. 1 is the food intakeevent detection subsystem 101. The role of this subsystem is to identifythe start and/or end of a food intake event and communicate an actual,probable or imminent occurrence of the start and/or end of a food intakeevent to other components in the system.

In general, the device detects what could be the start of a food intakeevent or the probable start of a food intake event, but the device wouldwork sufficient for its purposes so long as the device reasonablydetermines such start/probable start. For clarity, that detection isreferred to as a “deemed start” of a food intake event and when variousprocesses, operations and elements are to perform some action orbehavior in connection with the start of a food intake event, it wouldbe acceptable for those various processes, operations and elements totake a deemed start as the start even if occasionally the deemed startis not in fact a start of a food intake event.

In one embodiment, the detection and/or signaling of the occurrence ofthe deemed start of a food intake event coincides with the deemed startof a food intake event. In another embodiment, it may occur sometimeafter the deemed start of the food intake event. In yet anotherembodiment, it may occur sometime before the deemed start of the foodintake event. It is usually desirable that the signaling is close to thedeemed start of the food intake event. In some embodiments of thecurrent disclosure, it may be beneficial that the detection and/orsignaling of the deemed start of a food intake event occurs ahead of thestart of said food intake event. This may for example be useful if amessage or signal is to be sent to the user, a healthcare provider orcaregiver ahead of the start of the food intake event as a coachingmechanism to help steer a user's food intake decisions or eating habits.

In a preferred embodiment of the present disclosure, the detection ofthe start and/or ending of a food intake event by the food intake eventdetection subsystem 101 happens autonomously and does not require anyspecial user intervention. To accomplish this, the food intake eventdetection subsystem may use inputs 107 from one or more sensors 102.Sensors may include, but are not limited to, accelerometers, gyroscopes,magnetometers, magnetic angular rate and gravity (MARG) sensors, imagesensors, cameras, optical sensors, proximity sensors, pressure sensors,odor sensors, gas sensors, glucose sensors, Global Positioning Systems(GPS), and microphones.

Methods for autonomous detection may include, but are not limited to,detection based on monitoring of movement or position of the body or ofspecific parts of the body, monitoring of arm movement, position orgestures, monitoring of hand movement, position or gestures, monitoringof finger movement, position or gestures, monitoring of swallowingpatterns, monitoring of mouth and lips movement, monitoring of saliva,monitoring of movement of cheeks or jaws, monitoring of biting or teethgrinding, monitoring of signals from the mouth, the throat and thedigestive system. Methods for detection may include visual, audio or anyother types of sensory monitoring of the person and/or his or hersurroundings. The monitored signals may be generated by the dietarytracking and feedback system. Alternatively, they may be generated by aseparate system but be accessible to the dietary tracking and feedbacksystem through an interface. Machine learning and other data analyticstechniques may be applied to detect the start or probable start of afood intake event from the input signals being monitored.

In one example, the food intake detection system 101 may monitor theoutputs of accelerometer and/or gyroscope sensors to detect a possiblebite gesture or a possible sip gesture. Such gestures might bedetermined by a gesture processor that uses machine learning to distillgestures from sensor readings. The gesture processor might be part ofthe processor of the worn device or in another part of the system.

Gesture detection machine learning techniques as described elsewhereherein may be used to detect a bite gesture or sip gesture, but othertechniques are also possible. The food intake detection system 101 mayfurther assign a confidence level to the detected bite gesture or sipgesture. The confidence level corresponds to the likelihood that thedetected gesture is indeed a bite or sip gesture. The food intakedetection system may determine that the start of a food intake event hasoccurred based on the detection of a gesture and its confidence levelwithout any additional inputs. For example, the food intake eventdetection system 101 may decide that the start of a food intake eventhas occurred when the confidence level of the bite or sip gestureexceeds a pre-configured threshold.

Alternatively, when a possible bite or sip gesture has been detected,the food intake event detection system 101 may use additional inputs todetermine that the start or probable start of a food intake event hasoccurred. In one example, the food intake event detection system 101 maymonitor other gestures that are close in time to determine if the startof a food intake event has occurred. For example, upon detection of apossible bite gesture, the food intake event detection system 101 maywait for the detection of another bite gesture within a certain timewindow following the detection of the first gesture and/or with acertain confidence level before determining that the start of a foodintake event had occurred.

Upon such detection, the food intake detection system 101 may place oneor more circuits or components into a higher performance mode to furtherimprove the accuracy of the gesture detection. In another example, thefood intake event detection system 101 may take into consideration thetime of the day, or the location of the user to determine if the startor probable start of a food intake event has taken place. The foodintake event detection system may use machine learning or other dataanalytics techniques to improve the accuracy and reliability of itsdetection capabilities. For example, training data obtained from theuser and/or from other users at an earlier time may be used to train aclassifier. Training data may be obtained by asking for userconfirmation when a possible bite or sip gesture has been detected. Alabeled data record can then be created and stored in memory readable bythe gesture processor that includes the features related to the gesture,along with other contextual features, such as time of day or location. Aclassifier can then be trained on a labeled dataset comprised ofmultiple labeled data records set of labeled data records, and thetrained classifier model can then be used in a food intake eventdetection system to more accurately detect the start of a food intakeevent.

In another embodiment, the food intake detection subsystem may usetriggers to autonomously predict the probable start of a food intakeevent. Methods for autonomous detection of a probable start of a foodintake event based on triggers may include, but are not limited to,monitoring of a person's sleep patterns, monitoring of a person's stresslevel, monitoring of a person's activity level, monitoring of a person'slocation, monitoring of the people surrounding a person, monitoring of aperson's vital signs, monitoring of a person's hydration level,monitoring of a person's fatigue level. In some cases, the food intakedetection subsystem may monitor one or more specific trigger signals ortrigger events over a longer period of time and, in combination with thenon-real-time analysis and learning subsystem 105 apply machine learningor other data analytics techniques to predict the probable occurrence ofa start of a food intake event.

For example, without any additional information, it can be verydifficult to predict when a user will eat breakfast. However, if thesystem has a record over a number of days of the user's wake up time andthe day of the week, the system can use that historical pattern indetermining a likely time for the user to eat breakfast. Those recordsmight be determined by the system, possibly with feedback from the userabout their accuracy or those records might be determined by the userand input via a user interface of the system. The user interface mightbe the worn device itself or, for example, a smartphone app. As aresult, the system can process correlations in the historical data topredict the time or time window that the user is most likely to havebreakfast based on the current day of week and at what time the userwoke up. Other trigger signals or trigger events may also be used by thenon-real-time analysis and learning subsystem 105 to predict the timethat a user will eat breakfast.

In another example, the non-real-time analysis and learning system 105may, over a certain period of time record the stress level of a user.The stress level may, for example, be determined by monitoring andanalyzing the user's heart rate or certain parameters related to theuser's heart rate. The stress level may also be determined by analyzinga user's voice. The stress level may also be determined by analyzing thecontent of a user's messages or electronic communication. Other methodsfor determining the stress level are also possible. The non-real-timeanalysis and learning system 105 may furthermore, over the same periodof time, record the occurrence of food intake events and certaincharacteristics of the food intake event such as the pace of eating, thequantity of food consumed, the time spacing between food intake eventsetc. It may then be possible by analyzing the historical data of stresslevels, the occurrence of food intake events and food intake eventcharacteristics and by looking at correlations in the historical data ofstress levels, the occurrence of food intake events and food intakeevent characteristics, to predict based on the current stress level theprobability that a user will start a food intake event in a certain timewindow in the future, or predict what time window in the future, theuser will be most likely to start a food intake event. It may also bepossible to predict characteristics of said food intake event, such asfor example pace of eating or quantity of consumption.

In specific embodiments, the non-real time analysis and learningsubsystem may use historical data from different users, or a combinationof data from other users and from the wearer, and use similaritiesbetween one or more of the different users and the wearer, such as age,gender, medical conditions, etc. to predict the probable start of a foodintake event by the wearer.

In yet other examples, the non-real-time analysis and learning subsystem105 may use methods similar to the methods described herein to predictwhen a user is most likely to relapse in a binge eating episode or ismost likely to start convenience snacking.

A variety of sensors may be used for such monitoring. The monitoredsignals may be generated by the dietary tracking and feedback system.Alternatively, they may be generated by a separate system but beaccessible to the dietary tracking and feedback system for processingand/or use as trigger signals. Machine learning and other data analyticstechniques may also be applied to predict some other characteristics ofthe probable intake event, such as the type and/or amount of food thatwill likely be consumed, the pace at which a person will likely beeating, the level of satisfaction a person will have from consuming thefood etc.

The machine learning process performed as part of gesture recognitionmight use external data to further refine its decisions. This might bedone by non-real-time analysis and learning subsystem process. The dataanalytics process might, for example, consider the food intake eventsdetected by the gesture-sensing based food intake detection system andthe gesture-sensing based tracking and processing system, thus forming asecond layer of machine learning. For example, over a period of time,food intake events and characteristics related to those food intakeevents are recorded, such as eating pace, quantity of food consumption,food content, etc., while also tracking other parameters that are notdirectly, or perhaps not obviously, linked to the food intake event.This could be, for example, location information, time of day a personwakes up, stress level, certain patterns in a person's sleepingbehavior, calendar event details including time, event location andparticipant lists, phone call information including time, duration,phone number, etc., email metadata such as time, duration, sender, etc.The data analytics process then identifies patterns and correlations.For example, it may determine a correlation between the number ofcalendar events during the day and the characteristics of the foodintake event(s) in the evening. This might be due to the user being morelikely to start snacking when arriving home, or that dinner is largerand/or more rushed when the number of calendar event(s) for that dayexceeds a certain threshold. With subsystem 105, it becomes possible topredict food intake events and characteristics from other signals andevents that are not obviously linked to food intake.

Processing and analysis of one or more sensor inputs, and/or one or moreimages over longer periods of time, optionally using machine learning orother data analytics techniques may also be used to estimate theduration of a food intake event or may be used to predict that the endof a food intake event is probable or imminent.

In another embodiment, some user input 108 may be necessary or desirableto properly or more accurately detect the start and/or end of a foodintake event. Such user input may be provided in addition to externalinputs and inputs received from sensors 102. Alternatively, one or moreuser inputs may be used instead of any sensor inputs. User inputs mayinclude, but are not limited to activating a device, pressing a button,touching or moving a device or a specific portion of a device, taking apicture, issuing a voice command, making a selection on a screen orentering information using hardware and/or software that may include butis not limited to a keyboard, a touchscreen or voice-recognitiontechnology. If one or more user inputs are required, it is importantthat the user interaction is conceived and implemented in a way thatminimizes the negative impact on a person's normal activities or socialinteractions.

A food intake event detection subsystem may combine multiple methods toautonomously detect predict the actual, probably or imminent startand/or end of a food intake event.

Another component of the system is the tracking and processing subsystem103. In a preferred embodiment of the present disclosure, this subsysteminterfaces 109 with the food intake event detection subsystem 101, andgets activated when it receives a signal from the food intake eventdetection subsystem that the actual, probable or imminent start of anevent has been detected, and gets disabled when or sometime after itreceives a signal from the food intake event detection subsystem thatthe actual, probable or imminent ending of an event has been detected.Upon detection of the start of a food intake event, the device mighttrigger activation of other sensors or components of the food intaketracking system, and might also trigger the deactivation of those upondetection of the end of the food intake event.

In another embodiment of the current disclosure, the tracking andprocessing subsystem may be activated and/or deactivated independent ofany signals from the food intake detection subsystem. It is alsopossible that certain parameters be tracked and/or processedindependently of any signals from the food intake detection subsystem,whereas the tracking and/or processing of other parameters may only beinitiated upon receiving a signal from the food intake event detectionsubsystem.

The tracking and processing subsystem usually involves collecting dataover an interface 110 from one or more sensors 102 and processing thatdata to extract relevant information. The sensor inputs may be the sameor similar to the inputs sent to the food intake event detectionsubsystem. Alternatively, different and/or additional sensor inputs maybe collected. Sensors may include, but are not limited to,accelerometers, gyroscopes, magnetometers, image sensors, cameras,optical sensors, proximity sensors, pressure sensors, odor sensors, gassensors, Global Positioning Systems (GPS) circuit, microphones, galvanicskin response sensors, thermometers, ambient light sensors, UV sensors,electrodes for electromyographic (“EMG”) potential detection,bio-impedance sensors, spectrometers, glucose sensors, touchscreen orcapacitive sensors. Examples of sensor data include motion data,temperature, heart rate, pulse, galvanic skin response, blood or bodychemistry, audio or video recording and other sensor data depending onthe sensor type. The sensor inputs might be communicated to a processorwirelessly or via wires, in analog or digital form, intermediated bygating and/or clocking circuits or directly provided.

Processing methods used by the tracking and processing subsystem mayinclude, but are not limited to, data manipulation, algebraiccomputation, geo-tagging, statistical computing, machine learning,computer vision, speech recognition, pattern recognition, compressionand filtering.

Collected data may optionally be temporarily or permanently stored in adata storage unit 104. The tracking and processing subsystem 103 may useits interface 114 to the data storage unit 104 to place data or otherinformation in the data storage unit 104 and to retrieve data or otherinformation from the data storage unit 104.

In a preferred embodiment of the present disclosure, the collection ofdata, processing and tracking happen autonomously and do not require anyspecial user intervention. Tracked parameters may include, but are notlimited to, the following: location, temperature of surroundings,ambient light, ambient sounds, biometric information, activity levels,image captures of food, food names and descriptions, portion sizes,fluid intake, caloric and nutrient information, counts of mouthfuls,bite counts, sip counts, time durations between consecutive bites orsips, and duration of food intake events. Tracked parameters may alsoinclude, for each bite or sip, the time duration that the user's hand,arm and/or utensil is near the user's mouth, the time duration that thecontent of the bite or sip resides in the user's mouth beforeswallowing. The methods may vary based on what sensor data is available.

In other embodiments of the present disclosure, some user intervention111 is required or may be desirable to achieve for example greateraccuracy or input additional detail. User interventions 111 may include,but are not limited to, activating a device or specific functionality ofa device, holding a device in position, taking pictures, adding voiceannotations, recording video, making corrections or adjustments,providing feedback, doing data entry, taking measurements on food or onfood samples. Measurements may include, but are not limited to,non-destructive techniques such as for example obtaining one or morespectrographs of food items, or chemistry methods that may require asample taken from the food.

The processing of sensor data and user inputs by the tracking andprocessing subsystem 103 usually occurs real-time or near real-time.There may be some delays, for example to conserve power or to workaround certain hardware limitations, but in some embodiments, theprocessing occurs during the food intake event, or in case of trackingoutside of a food intake event, around the time that the sensor or userinputs have been received.

In certain implementations or under certain circumstances, there may notbe real-time or near real-time access to the processing unit required toperform some or all of the processing. This may, for example, be due topower consumption or connectivity constraints. Other motivations orreasons are also possible. In that case, the inputs and/or partiallyprocessed data may be stored locally until a later time when access tothe processing unit becomes available.

In one specific embodiment of the present disclosure, sensor signalsthat track movement of a person's arm, hand or wrist may be sent to thetracking and processing subsystem. The tracking and processing subsystemmay process and analyze such signals to identify that a bite of food orsip of liquid has been consumed or has likely been consumed by saidperson. The tracking and processing subsystem may furthermore processand analyze such signals to identify and/or quantify other aspects ofeating behavior such as for example the time separation between bites orsips, the speed of hand-to-mouth movement etc. The tracking andprocessing subsystem may furthermore process and analyze such signals toidentify certain aspects of the eating method such as, for example,whether the person is eating with a fork or spoon, is drinking from aglass or can, or is consuming food without using any utensils.

In a specific example, it might be that the wearer rotates his or herwrist in one direction when bringing an eating utensil or hand to themouth when taking a bite, but rotates in the other direction whensipping a liquid. The amount of rotation of a wearer's wrist as he orshe moves his or her wrist to the mouth or away from the mouth and theduration that the wrist is held at a higher rotation angle may also bedifferent for a drinking gesture versus an eating gesture. Other metricsmay be used to distinguish eating gestures from drinking gestures or todistinguish differences in eating methods. A combination of multiplemetrics may also be used. Other examples of metrics that may be used todistinguish eating gestures from drinking gestures or to distinguishdifferences in eating methods include but are not limited to the changein angle of the roll from the start or approximate start of the gestureuntil the time or approximate time that the hand reaches the mouth, thechange in angle of the roll from the time or approximate time that thehand is near the mouth until the end or approximate end of the gesture,the variance of accelerometer or gyroscope readings across one or moreof the axes for a duration of time when the hand is near the mouth, orfor a duration of time that is centered around when the hand is near themouth, or for a duration of time that may not be centered around whenthe hand is near the mouth but that includes the time when the hand isthe nearest to the mouth, the variance of the magnitude of theaccelerometer readings for a duration of time when the hand is near themouth, or for a duration of time that is centered around when the handis the nearest to the mouth, or for a duration of time that may not becentered around when the hand is the nearest to the mouth but thatincludes the time when the hand is the nearest to the mouth, the maximumvalue of the magnitude of the accelerometer readings for a duration oftime when the hand is near the mouth, or for a duration of time that iscentered around when the hand is the nearest to the mouth, or for aduration of time that may not be centered around when the hand is thenearest to the mouth but that includes the time when the hand is thenearest to the mouth. The magnitude of the accelerometer reading may bedefined as square root of the acceleration in each orthogonal direction(e.g., sense acceleration in the x, y, and z directions and calculateSQRT(a_(x) ²+a_(y) ²+a_(z) ²)).

The position of the hand vis-á-vis the mouth can, for example, bedetermined by monitoring the pitch or the worn device and from there thepitch of the wearer's arm. The time corresponding to the peak of thepitch could be used as the moment in time when the hand is the nearestto the mouth. The time when the pitch starts rising could, for example,be used as the start time of the gesture. The time when the pitch stopsfalling could for example be used as the end time of the gesture.

Other definitions for nearest mouth position, start of movement and endof movement are also possible. For example, the time when the rollchanges direction could be used instead to determine the time when thearm or hand is the nearest to the mouth. The time when the roll stopschanging in a certain direction or at a certain speed could be usedinstead to determine the start time of the movement towards the mouth.

The tracking and processing subsystem may furthermore process andanalyze such signals to determine appropriate or preferred times toactivate other sensors. In one specific example, the tracking andprocessing subsystem may process and analyze such signals to determinean appropriate or preferred time to activate one or more cameras to takeone or more still or moving images of the food. By leveraging sensorsthat track arm, hand, finger or wrist movement and/or the orientationand position of the camera to activate the camera and/or automate theimage capture process, the complexity, capabilities and powerconsumption of the image-capture and image analysis system can begreatly reduced, and in certain cases better accuracy may be achieved.It also significantly reduces any privacy invasion concerns, as it nowbecomes possible to more precisely control the timing of image capturingand make it coincide with the cameras being focused on the food.

For example, the processor might analyze motion sensor inputs from anaccelerometer, a gyroscope, a magnetometer, etc., to identify theoptimal time to activate camera and capture picture and trigger thecamera at that time, perhaps based on when the processor determines thatthe view region of the camera encompasses the food to be photographed.In one example, the processor determines the start of an eating eventand signals the wearer to capture an image of the food being eaten andalso determines the end of the eating event and again signals the wearerto capture an image of what remains of the food or the plate, etc. Suchimages can be processed to determine consumption amounts and/or toconfirm consumption amounts already determined by the processor. In someembodiments, the image processing can be used as part of feedback totrain machine learning that the processor uses.

In some embodiments, the system may use sensors that track the movementof the wearer's arm or hand and only activate the camera when the systemdetermines from the movement sensing that the arm or hand are near themouth. In another example, the system may activate the camera sometimebetween the start of the movement towards the mouth and the time whenthe arm or hand is the nearest to the mouth. In yet another example, thesystem may activate the camera sometime between the time when the arm orhand is the nearest to the mouth and the end of the movement away fromthe mouth.

As mentioned above, the position of the hand vis-á-vis the mouth can bedetermined by monitoring the pitch and a rising pitch indicating a starttime of a movement towards the mouth and a falling pitch indicating anend time. Other definitions for nearest mouth position, start ofmovement and end of movement are also possible.

The position of the hand vis-á-vis the mouth can, for example, bedetermined by monitoring the pitch or the worn device and from there thepitch of the wearer's arm. The time corresponding to the peak of thepitch could be used as the moment in time when the hand is the nearestto the mouth. The time when the pitch starts rising could, for example,be used as the start time of the gesture. The time when the pitch stopsfalling could for example be used as the end time of the gesture.

The processing and analysis of sensor signals that track movement of auser's arm, hand or wrist may be combined with other methods such as theimage capture of food as it enters the mouth as proposed to build inredundancy and improve the robustness of a dietary tracking and feedbacksystem. For example, by processing and analysis of a user's arm, hand orwrist movement, information related to bite count and bite patternswould still be preserved, even if the camera were to be obscured ortampered with.

One or more of the sensor inputs may be still or streaming imagesobtained from one or more camera modules. Such images may require somelevel of processing and analysis. Processing and analysis methods may,among other methods, include one or more of the following methods:compression, deletion, resizing, filtering, image editing, and computervision techniques to identify objects such as, for example, specificfoods or dishes, or features such as, for example, portion sizes.

In addition to measuring bite counts and sip counts, the processor mightanalyze specifics, such as cadence and duration, to determine bite andsip sizes. Measuring the time that the wearer's hand, utensil or fluidcontainer was near their mouth might be used to derive a “near-mouth”duration that is in turn used as an input to generate an estimate sizeof the bite or sip. The amount of rotation of the wrist when sippingmight be useful for hydration tracking.

Measuring the amount of rotation of the wrist in one or more timesegments that are within the start and the end of the gesture may alsobe used to estimate the size of the bite or sip. For example, a systemmay measure the amount of rotation of the wrist from a time sometimeafter the start of the gesture to the time when the arm or hand is thenearest to the mouth. The time corresponding to the peak of the pitchcould be used as the moment in time when the hand is the nearest to themouth. The time when the pitch starts rising could for example be usedas the start time of the movement towards the mouth. The time when thepitch stops falling could for example be used as the end time of themovement away from the mouth. Other definitions for nearest mouthposition, start of movement and end of movement are also possible. Forexample, the time when the roll changes direction could be used insteadas the time when the arm or hand is the nearest to the mouth. The timewhen the roll stops changing in a certain direction or at a certainspeed could be used as the start time of the movement towards the mouth.One or more still or streaming images may be analyzed and/or compared bythe tracking and processing subsystem for one or multiple purposesincluding, but not limited to, the identification of food items, theidentification of food content, the identification or derivation ofnutritional information, the estimation of portion sizes and theinference of certain eating behaviors and eating patterns.

As one example, computer vision techniques, optionally combined withother image manipulation techniques may be used to identify foodcategories, specific food items and/or estimate portion sizes.Alternatively, images may be analyzed manually using a Mechanical Turkprocess or other crowdsourcing methods. Once the food categories and/orspecific food items have been identified, this information can be usedto retrieve nutritional information from one or more foods/nutritiondatabases.

As another example, information about a user's pace of eating ordrinking may be inferred from analyzing and comparing multiple imagescaptured at different times during the course of a food intake event. Asyet another example, images, optionally combined with other sensorinputs, may be used to distinguish a sit-down meal from finger foods orsnacks. As yet another example, the analysis of one image taken at thestart of a food intake event and another image taken at the end of afood intake event may provide information on the amount of food that wasactually consumed.

In a general case, sensor data is taken in by a processor that analyzesthat sensor data, possibly along with prior recorded data and/ormetadata about a person about whom the sensor data is sensing. Theprocessor performs computations, such as those described herein, toderive a sequence of sensed gestures. A sensed gesture might be one ofthe gestures described elsewhere herein, along with pertinent data aboutthe sensed gesture, such as the time of occurrence of the sensedgesture. The processor analyzes the sequence of sensed gestures todetermine the start of a behavior event, such as the starting of aneating event.

The determination of the start of an eating event may be based on asequence of sensed gestures, but it may also be based on the detectionof a single event (possibly with non-gesture based context). Forexample, if the system detects a bite gesture with a reasonably highconfidence level, the processor might consider that detection of thatindividual gesture to be the start of an eating event. The processor canalso analyze the sequence of sensed gestures to determine the end of thebehavior event. The determination of the end of an eating event may alsobe based on the absence of detected events. For example, if no bitegestures are detected in a given time period, the processor can assumethat the eating event ended.

Knowing the start and end of a behavior event allows the processor tomore accurately determine the gestures, since they are taken in contextand/or the processor may enable additional sensors or place one or moresensors or other components in a higher performance state, such as inexamples described elsewhere herein. Knowing the start and end of abehavior event also allows for power savings as, in some cases, it maybe possible to place the worn device in a lower power mode outsidecertain behavior events. Also, aggregation of individual gestures intoevents, possibly combined with prior recorded data about similarbehavior events from the same user or from other users in the past,allows the processor to derive meaningful characteristics about thebehavior event. For example, an eating pace during breakfast, lunch,dinner can be determined in this manner. As another example, if theprocessor has a state for a current behavior and that current behavioris teeth brushing, gestures that might appear to be eating or drinkinggestures would not be interpreted as eating or drinking gestures andthus not interpret sipping while teeth brushing as being consumption ofliquids. Behavior events might be general events (eating, walking,brushing teeth, etc.) or more specific (eating with a spoon, eating witha fork, drinking from a glass, drinking from a can, etc.).

While it might be possible to decode an indirect gesture, such asdetecting a pointing gesture and then determining the object that thesensed person is pointing at, of interest are gestures that themselvesare directly part of the event being detected. Some gestures areincidental gestures, such as gestures associated with operating thedevice, in which case incidental gestures might be excluded fromconsideration.

In a specific example, the system uses some set of sensors to determinethe start of an eating event with some confidence level and if theconfidence level is higher than a threshold, the system activatesadditional sensors. Thus, the accelerometer sensor might be used todetermine the start of an eating event with high confidence level, but agyroscope is put in a low power mode to conserve battery life. Theaccelerometer alone can detect a gesture that is indicative of aprobable bite or sip (e.g., an upward arm or hand movement or a hand orarm movement that is generally in the direction of the mouth), or agesture that is generally indicative of the start of an eating event.Upon detection of a first gesture that is generally indicative of apossible start of an eating event, the additional sensors (e.g.,gyroscope, etc.) may then be enabled. If a subsequent bite or sipgesture is detected, the processor determines that the start of aneating event had occurred and with a higher confidence level.

Knowing the start/end of a behavior event allows the processor to placeone or more sensor or other components in a higher performance state forthe duration of the behavior event. For example, when a start of abehavior event has been determined, the processor may increase thesampling rate of the accelerometer and/or gyroscope sensors used todetect gestures. As another example, when a start of a behavior eventhas been determined, the processor may increase the update rate at whichsensor data are sent to electronic device 19 for further processing toreduce latency.

Referring again to FIG. 1, in addition to the tracking and processingsubsystem, the system of FIG. 1 may also include a non-real-timeanalysis and learning subsystem 105. The non-real-time analysis andlearning subsystem can perform an analysis on larger datasets that takea longer time to collect, such as historical data across multiple foodintake events and/or data from a larger population. Methods used by thenon-real-time analysis and learning subsystem may include, but are notlimited to, data manipulation, algebraic computation, geo-tagging,statistical computing, machine learning and data analytics, computervision, speech recognition, pattern recognition, compression andfiltering.

Methods used by non-real-time analysis and learning subsystem 105 may,among other things, include data analytics on larger sets of datacollected over longer periods of time. As an example, one or more datainputs may be captured over a longer period of time and across multiplefood intake events to train a machine learning model. Such data inputsare hereafter referred to as training data sets. It is usually desirablethat the period of time over which a training data set is collected,hereafter referred to as the training period, is sufficiently long suchthat the collected data is representative of a person's typical foodintake.

A training data set may, among other things, include one or more of thefollowing food intake related information: number of bites per foodintake event, total bites count, duration of food intake event, pace offood intake or time between subsequent counts, categorization of foodintake content such as for example distinguishing solid foods fromliquids or sit-down meals from snacks or finger-foods. This informationmay be derived from one or more sensor inputs.

A training data set may furthermore include images of each or most itemsthat were consumed during each of the food intake events within thetraining period. The images may be processed using computer visionand/or other methods to identify food categories, specific food itemsand estimate portion sizes. This information may then in turn be used toquantify the number of calories and/or the macro-nutrient content of thefood items such as amounts of carbohydrates, fat, protein, etc.

In case the food was not consumed in its entirety, it may be desirableto take one picture of the food item at the start of the food intakeevent and one picture at the end of the food intake event to derive theportion of the food that was actually consumed. Other methods including,but not limited to, manual user input, may be used to add portion sizeinformation to the data in a training data set.

A training data set may furthermore include meta-data that do notdirectly quantify the food intake and/or eating behavior and patterns,but that may indirectly provide information, may correlate with foodintake events and/or eating behavior and/or may be triggers for theoccurrence of a food intake event or may influence eating habits,patterns and behavior. Such meta-data may, among other things, includeone or more of the following: gender, age, weight, social-economicstatus, timing information about the food intake event such as date,time of day, day of week, information about location of food intakeevent, vital signs information, hydration level information, and otherphysical, mental or environmental conditions such as for example hunger,stress, sleep, fatigue level, addiction, illness, social pressure, andexercise.

One or more training data sets may be used to train one or more machinelearning models which may then be used by one or more components of thedietary tracking and feedback systems to predict certain aspects of afood intake event and eating patterns and behaviors.

In one example, a model may be trained to predict the occurrence of afood intake event based on the tracking of one or more meta-data thatmay influence the occurrence of a food intake event. Othercharacteristics related to the probable food intake event, such as thetype and/or amount of food that will likely be consumed, the pace atwhich a person will likely be eating, the duration of the food intakeevent, and/or the level of satisfaction a person will have fromconsuming the food may also be predicted. Meta-data may, among otherthings, include one or more of the following: gender, age, weight,social-economic status, timing information about the food intake eventsuch as date, time of day, day of week, information about location offood intake event, vital signs information, hydration level information,and other physical, mental or environmental conditions such as forexample hunger, stress, sleep, fatigue level, addiction, illness, socialpressure, and exercise.

In another example, machine learning and data analytics may be appliedto derive metrics that may be used outside the training period toestimate caloric or other macro-nutrient intake, even if only limited orno food intake sensor inputs or images are available. Meta-data may beused to further tailor the value of such metrics based on additionalcontextual information. Meta-data may, among other things, include oneor more of the following: gender, age, weight, social-economic status,timing information about the food intake event such as date, time ofday, day of week, information about location of food intake event,information about generic food category, vital signs information,hydration level information, calendar events information, phone calllogs, email logs, and other physical, mental or environmental conditionssuch as for example hunger, stress, sleep, fatigue level, addiction,illness, social pressure, and exercise.

One example of such a metric would be “Calories per Bite”. By combiningthe bites count with the caloric information obtained from imageprocessing and analysis, a “Calories per bite” metric can be establishedfrom one or more training data sets. This metric can then be usedoutside the training period to estimate caloric intake based on bitescount only, even if no images or only limited images are available.

Another metric could be “Typical Bite Size”. By combining the bitescount with the portion size information obtained from image processingand analysis, a “Typical Bite size” metric can be established from oneor more training data sets. This metric can then be used outside thetraining period to estimate portion sizes based on bites count only,even if no images or only limited images are available. It may also beused to identify discrepancies between reported food intake and measuredfood intake based on bite count and typical bite size. A discrepancy mayindicate that a user is not reporting all the food items that he or sheis consuming. Or, alternatively, it may indicate that a user did notconsume all the food that he or she reported.

Bite actions might be determined by a processor reading accelerometerand gyroscope sensors, or more generally by reading motion sensors thatsense movement of body parts of the wearer. Then, by counting bites, atotal number of bites can be inferred. Also, the time sequence of thebites can be used by the processor do deduce an eating pattern.

Non-real-time analysis and learning subsystem 105 may also be usedtrack, analyze and help visualize larger sets of historical data, trackprogress against specific fixed or configured goals, and help establishsuch goals. It may furthermore be used to identify and track records,streaks and compare performance with that of friends or larger,optionally anonymous, populations.

Furthermore, in certain embodiments, non-real-time analysis and learningsubsystem 105 may among other data manipulation and processingtechniques, apply machine learning and data analytics techniques topredict the imminence of or the likelihood of developing certain healthissues, diseases and other medical conditions. In this case, trainingtypically requires historical food intake and/or eating behaviors datacaptured over longer periods of time and across a larger population. Itis furthermore desirable that training data sets include additionalmeta-data such as age, weight, gender, geographical information,socio-economic status, vital signs, medical records information,calendar information, phone call logs, email logs and/or otherinformation. Predictions may in turn be used to help steer healthoutcomes and/or prevent or delay the onset of certain diseases such asfor example Diabetes.

Non-real-time and learning subsystem 105 may also be used to learn andextract more information about other aspects including, but not limitedto, one or more of the following: a user's dietary and food preferences,a user's dining preferences, a user's restaurant preferences, and auser's food consumption. Such information may be used by the food intaketracking and feedback system to make specific recommendations to user.The food intake tracking and feedback system described in herein mayalso interface to or be integrated with other systems such as restaurantreservation systems online food or meal ordering systems, and others tofacilitate, streamline or automate the process of food or meal orderingor reservations.

Non-real-time and learning subsystem 105 may also be used to monitorfood intake over longer periods of times and detect any unusually longepisodes of no food intake activity. Such episodes may, among otherthings, indicate that the user stopped using the device, intentional orunintentional tampering with the device, a functional defect of thedevice or a medical situation such as for example a fall or death orloss of consciousness of the user. Detection of unusually long episodesof no food intake activity may be used to send a notification or alertto the user, one or more of his caregivers, a monitoring system, anemergency response system, or to a third party who may have a direct orindirect interest in being informed about the occurrence of suchepisodes.

Another component of the system shown in FIG. 1 is the feedbacksubsystem 106. The feedback subsystem 106 provides one or more feedbacksignals to the user or to any other person to which such feedbackinformation may be relevant. The feedback subsystem 106 may providereal-time or near real-time feedback related to a specific food intakeevent. Real-time or near real-time feedback generally refers to feedbackgiven around the time of a food intake event. This may include feedbackgiven during the food intake event, feedback given ahead of the start ofa food intake event and feedback given sometime after the end of a foodintake event. Alternatively, or additionally, the feedback subsystem mayprovide feedback to the user that is not directly linked to a specificfood intake event.

Feedback methods used by the feedback subsystem may include, but are notlimited to, haptic feedback whereby a haptic interface is used thatapplies forces, vibrations and/or motion to the user, audio feedbackwhere a speaker or any other audio interfaces may be used, or visualfeedback whereby a display, one or more LEDs and/or projected lightpatterns may be used. The feedback subsystem may use only one or acombination of more than one feedback method.

The feedback subsystem may be implemented in hardware, in software or ina combination of hardware and software. The feedback subsystem may beimplemented on the same device as the food intake event detectionsubsystem 101 and/or the tracking and processing subsystem 103.Alternatively, the feedback subsystem may be implemented in a devicethat is separate from the food intake event detection subsystem 101and/or the tracking and processing subsystem 103. The feedback subsystemmay also be distributed across multiple devices, some of which mayoptionally house portions of some of the other subsystems illustrated inFIG. 1.

In one embodiment, the feedback subsystem may provide feedback to theuser to signal the actual, probable or imminent start of a food intakeevent. The feedback subsystem may also provide feedback to the userduring a food intake event to remind the user of the fact that a foodintake event is taking place, to improve in-the-moment awareness and/orto encourage mindful eating. The feedback subsystem may also provideguidance on recommended portion sizes and/or food content, or providealternative suggestions to eating. Alternative suggestions may bedefault suggestions or it may be custom suggestions that have beenprogrammed or configured by the user at a different time.

Feedback signals may include, but are not limited to, periodic hapticfeedback signals on a wearable device, sound alarms, display messages,or one or more notifications being pushed to his or her mobile phonedisplay.

Upon receiving a signal that indicates the start of a food intake event,or sometime thereafter, the user may confirm that a food intake event isindeed taking place. Confirmation can be used to for example triggerlogging of the event or may cause the system to prompt the user foradditional information.

In another embodiment of the present disclosure, the feedback subsysteminitiates feedback during a food intake event only if a certainthreshold of one or more of the parameters being tracked is reached. Asan example, if the time between subsequent bites or sips is beingtracked, feedback to the user may be initiated if the time, possiblyaveraged over a multiple bites or sips, is shorter than a fixed orprogrammed value to encourage the user to slow down. Similarly, feedbackmay be initiated if a fixed or programmed bites or sips count is beingexceeded.

In feedback subsystems where feedback is provided during a food intakeevent, the feedback provided by the feedback subsystem usually relatesto specifics of that particular food intake event. However, otherinformation including, but not limited to, information related to priorfood intake events, biometric information, mental health information,activity or fitness level information, and environmental information mayalso be provided by the feedback subsystem.

In yet another embodiment of the present disclosure, the feedbacksubsystem may be sending one or more feedback signals outside a specificfood intake event. In one example of such an embodiment, ambienttemperature and/or other parameters that may influence hydrationrequirements or otherwise directly or indirectly measure hydrationlevels may be tracked. Such tracking may happen continuously orperiodically, or otherwise independent from a specific food intakeevent. If one or more such parameters exceed a fixed or programmedthreshold, a feedback signal may be sent to for example encouragehim/her to take measures to improve hydration. The feedback subsystemmight evaluate its inputs and determine that a preferred time forsending feedback is not during a food intake event, but after the foodintake event has ended. Some of the inputs to the feedback subsystemmight be from a food intake event, but some might be from othermonitoring not directly measured as a result of the food intake event.

The decision to send a feedback signal may be independent of any foodintake tracking, such as in the embodiment described in the previousparagraph. Alternatively, such a decision may be linked to food intaketracking across one or multiple food intake events. For example, in oneembodiment of the current disclosure, the system described above couldbe modified to also track, either directly or indirectly, a person'sintake of fluids. For different ambient temperature ranges, saidembodiment could have pre-programmed fluid intake requirementthresholds. If for a measured ambient temperature, a person's intake offluids, possibly tracked and accumulated over a certain period of time,is not meeting the threshold for said ambient temperature, the systemmay issue a feedback signal to advise said person to increase his or herlevels of fluid intake.

Similarly, feedback signals or recommendations related to food intakemay among other parameters, be linked to tracking of activity levels,sleep levels, social context or circumstances, health or diseasediagnostics, and health or disease monitoring.

In yet another embodiment of the current disclosure, the feedbacksubsystem may initiate a feedback signal when it has detected that afood intake event has started or is imminent or likely. In such anembodiment, feedback could for example be used as a cue to remind theuser log the food intake event or certain aspects of the food intakeevent that cannot be tracked automatically, or to influence or steer aperson's food intake behavior and/or the amount or content of the foodbeing consumed.

Information provided by the feedback subsystem may include but is notlimited to information related to eating patterns or habits, informationrelated to specific edible substances, such as for example the name, thedescription, the nutrient content, reviews, ratings and/or images offood items or dishes, information related to triggers for food intake,information related to triggers for eating patterns or habits, biometricor environmental information, or other information that may be relevanteither directly or indirectly to a person's general food intakebehavior, health and/or wellness.

The feedback subsystem may include the display of images of food itemsor dishes that have been consumed or may be consumed. Furthermore, thefeedback subsystem may include additional information on said food itemsor dishes, such as for example indication of how healthy they are,nutrient content, backstories or preparation details, ratings,personalized feedback or other personalized information.

In certain embodiments of the current disclosure, the informationprovided by the feedback subsystem may include non-real-timeinformation. The feedback subsystem may for example include feedbackthat is based on processing and analysis of historical data and/or theprocessing and analysis of data that has been accumulated over a largerpopulation of users. The feedback subsystem may further provide feedbackthat is independent of the tracking of any specific parameters. As anexample, the feedback subsystem may provide generic food, nutrition orhealth information or guidance.

In certain embodiments of the current disclosure, the user may interactwith the feedback subsystem and provide inputs 116. For example, a usermay suppress or customize certain or all feedback signals.

Non-real time feedback may, among other things, include historical data,overview of trends, personal records, streaks, performance against goalsor performance compared to friends or other people or groups of people,notifications of alarming trends, feedback from friends, social networksand social media, caregivers, nutritionists, physicians etc., coachingadvice and guidance.

Data or other information may be stored in data storage unit 104. It maybe stored in raw format. Alternatively, it may be stored after it hasbeen subject to some level of processing. Data may be stored temporarilyor permanently. Data or other information may be stored for a widevariety of reasons including, but not limited to, temporary storagewhile waiting for processor or other system resources to becomeavailable, temporary storage to be combined with other data that may notbe available until a later time, storage to be fed back to the user inraw or processed format through the feedback subsystem, storage forlater consultation or review, storage for analysis for dietary and/orwellness coaching purposes, storage for statistical analysis across alarger population or on larger datasets, storage to perform patternrecognition methods or machine learning techniques on larger datasets.

The stored data and information, or portions thereof, may be accessibleto the user of the system. It is also possible that the stored data andinformation or portions thereof, may be shared with or can be accessedby a third party. Third parties may include, but are not limited to,friends, family members, caregivers, healthcare providers,nutritionists, wellness coaches, other users, companies that developand/or sell systems for dietary tracking and coaching, companies thatdevelop and/or sell components or subsystems for systems for dietarytracking and coaching, and insurance companies. In certaincircumstances, it may be desirable that data is made anonymous beforemaking it available to a third party.

FIG. 2 illustrates some of the components disposed in an electronicsystem used for dietary tracking and coaching, in accordance with oneembodiment of the present disclosure. The electronic system includes afirst electronic device 218, a second electronic device 219 and acentral processing and storage unit 220. A typical system might have acalibration functionality, to allow for sensor and processorcalibration.

Variations of the system shown in FIG. 2 are also possible and areincluded in the scope of the present disclosure. For example, in onevariation, electronic device 218 and electronic device 219 may becombined into a single electronic device. In another variation, thefunctionality of electronic device 218 may be distributed acrossmultiple devices. In some variations, a portion of the functionalityshown in FIG. 2 as being part of electronic device 218 may instead beincluded in electronic device 219. In some other variations, a portionof the functionality shown in FIG. 2 as being part of electronic device219 may instead be included in electronic device 218 and/or centralprocessing and storage unit 220. In yet another variation, the centralprocessing and storage unit may not be present and all processing andstorage may be done locally on electronic device 218 and/or electronicdevice 219. Other variations are also possible.

An example of the electronic system of FIG. 2 is shown in FIG. 3.Electronic device 218 may for example be a wearable device 321 that isworn around the wrist, arm or finger. Electronic device 218 may also beimplemented as a wearable patch that may be attached to the body or maybe embedded in clothing. Electronic device 218 may also be a module oradd-on device that can for example be attached to another wearabledevice, to jewelry, or to clothing. Electronic device 219 may forexample be a mobile device 322 such as a mobile phone, a tablet or asmart watch. Other embodiments of electronic device 219 and ofelectronic device 218 are also possible. The central processing andstorage unit 220 usually comprises of one or more computer systems orservers and one or more storage systems. The central processing andstorage unit 220 may for example be a remote datacenter 324 that isaccessible via the Internet using an Internet connection 325. Thecentral processing and storage unit 220 is often times shared amongand/or accessed by multiple users.

The wearable device 321 may communicate with mobile device 322 over awireless network. Wireless protocols used for communication over awireless network between wearable device 321 and mobile device 322 mayinclude, but is not limited to, Bluetooth, Bluetooth Smart (a.k.a.Bluetooth Low Energy), Bluetooth Mesh, ZigBee, Wi-Fi, Wi-Fi Direct, NFC,Cellular and Thread. A proprietary or wireless protocol, modificationsof a standardized wireless protocol or other standardized wirelessprotocols may also be used. In another embodiment of the currentdisclosure, the wearable device 321 and the mobile device 322 maycommunicate over a wired network.

The mobile device 322 may communicate wirelessly with a base station orAccess Point (“AP”) 323 that is connected to the Internet via Internetconnection 325. Via the Internet connection 325, mobile device 322 maytransfer data and information from wearable device 321 to one or morecentral processing and storage unit 220 that reside at a remotelocation, such as for example a remote data center. Via Internetconnection 325, mobile device 322 may also transfer data and informationfrom one or more central processing and storage unit 220 that reside ata remote location to wearable device 321. Other examples are alsopossible. In some embodiments, the central processing and storage unit220 may not be at a remote location, but may reside at the same locationor close to the wearable device 321 and/or mobile device 322. Wirelessprotocols used for communication between the mobile device 322 and thebase station or access point 323 may be the same as those between themobile device and the wearable device. A proprietary or wirelessprotocol, modifications of a standardized wireless protocol or otherstandardized wireless protocols may also be used.

The electronic system of FIG. 2 may also send data, information,notifications and/or instructions to and/or receive data, information,notifications and/or instructions from additional devices that areconnected to the Internet. Such devices could for example be a tablet,mobile phone, laptop or computer of one or more caregivers, members ofthe physician's office, coaches, family members, friends, people whomthe user has connected with on social media, or other people to whom theuser has given the authorization to share information. One example ofsuch a system is shown in FIG. 4. In the example shown in FIG. 4,electronic device 441 is wirelessly connected to base station or AccessPoint 440 that is connected to the Internet via Internet connection 442.Examples of electronic device 441 may include, but are not limited to, atablet, mobile phone, laptop, computer, or smart watch. Via Internetconnection 442, electronic device 441 may receive data, instructions,notifications or other information from one or more central processingand storage units that may reside locally or at a remote location, suchas for example a remote data center. The communication capability caninclude Internet connection 442 or other communication channels.Electronic device 441 may also send information, instructions ornotifications to one or more computer servers or storage units 439.Central processing and storage unit 439 may forward this information,instructions or notifications to mobile device 436 via the Internet 438and the base station or Access Point (“AP”) 437.

Other examples are also possible. In some embodiments, the centralprocessing and storage unit 439 may not be at a remote location, but mayreside at the same location or close to the wearable device 435 and/ormobile device 436. FIG. 4 shows electronic device 441 as beingwirelessly connected to the base station or Access Point. A wiredconnection between electronic device 441 and a router that connects tothe Internet via an Internet connection 442 is also possible.

FIG. 5 illustrates another embodiment of the present disclosure. In FIG.5, a wearable device 543 can exchange data or other information directlywith a central processing and storage system 546 via a base station orAccess Point 544 and the Internet without having to go through mobiledevice 545. Mobile device 545 may exchange data or other informationwith wearable device 543 either via central processing and storagesystem 546 or via a local wireless or wired network. The centralprocessing and storage system 546 may exchange information with one ormore additional electronic devices 550.

FIG. 6 illustrates some of the components disposed in electronic device218, in accordance with one embodiment. Electronic device 218 typicallyincludes, in part, one or more sensor units 627, a processing unit 628,memory 629, a clock or crystal 630, radio circuitry 634, and a powermanagement unit (“PMU”) 631. Electronic device 218 may also include oneor more camera modules 626, one or more stimulus units 633 and one ormore user interfaces 632. Although not shown, other components likecapacitors, resistors, inductors may also be included in said electronicdevice 218. Power Management unit 631 may, among other things, includeone or more of the following: battery, charging circuitry, regulators,hardware to disable the power to one or more components, power plug.

In many embodiments, electronic device 218 is a size constrained,power-sensitive battery operated device with a simple and limited userinterface. Where power is limited, electronic device 218 might beprogrammed to save power outside of behavior events. For example, aprocessor in electronic device 218 might be programmed to determine thestart of a behavior event, such as an eating event, and then power upadditional sensors, place certain sensors in a higher performance modeand/or perform additional computations until the processor determines anend of the behavior event, at which point the processor might turn offthe additional sensors, place certain sensors back in a lowerperformance mode and omit the additional computations.

For example, the processor might be programmed to disable allmotion-detection related circuitry, with exception of an accelerometer.The processor could then monitor accelerometer sensor data and if thosedata indicate an actual or prominent food intake activity such as a biteor sip gesture, then the processor could activate additional circuitry,such as a data recording mechanism. The processor might use theaccelerometer sensor data to monitor a pitch of the wearer's arm.

For example, the processor might measure pitch of the wearer's arm untilthe pitch exceeds a certain threshold, perhaps one indicative of a handor arm movement towards the wearers' mouth. Once that is detected, theprocessor can change the state (such as by changing a memory locationset aside for this state from “inactive” or “out-of-event” to “in anaction” or “in-event”) and activate additional circuitry or activate ahigher performance mode of specific circuitry or components. In anotherembodiment, other accelerometer sensor data characteristics such asfirst integral of acceleration (velocity) or the second integral ofacceleration (distance traveled), or characteristics related to orderived from the first and/or second integral of acceleration might beused, as determined from one or more accelerometer axis. A machinelearning process might be used to detect specific movements andtranslate those to gestures.

An end of a food intake event might be detected by the processor byconsidering whether a certain time has expired since a last bite or sipmovement or when other data (metadata about the wearer, motion-detectionsensor data, and/or historical data of the wearer, or a combination ofthose). Based on those, the processor makes a determination that a foodintake event is not likely and then changes the state of the electronicdevice to an inactive monitoring state, possibly a lower power mode.

The lower power mode might be implemented by the processor reducing thesampling rate of the accelerometer and/or gyroscope, powering down thegyroscope, reducing the update rate at which sensor data is transferredfrom the electronic device (such as electronic device 218) to thesupport device (such as electronic device 219), compressing the databefore transferring the data from the sensing electronic device to thesupport electronic device.

In some embodiments of the present disclosure, some of the componentsthat are shown in FIG. 5 as separate components may be combined. As anexample, the processing unit, memory, radio circuitry and PMUfunctionality may entirely or in part be combined in a single wirelessmicrocontroller unit (“MCU”). Other combinations are also possible.Similarly, components that are shown as a single component in FIG. 5 maybe implemented as multiple components. As an example, the processingfunctionality may be distributed across multiple processors. Likewise,data storage functionality may be distributed across multiple memorycomponents. Other examples of distributed implementations are alsopossible.

In another embodiment of the present disclosure, the radio circuitry maynot be present and instead a different interface (such as for example aUSB interface and cable) may be used to transfer data or information toand/or from the electronic device 218.

Stimulus unit 633 may provide feedback to the user of the electronicdevice. A stimulus unit 633 may include but is not limited to a hapticinterface that applies forces, vibrations or motions to the user, aspeaker or headphones interface that provides sounds to the user, and adisplay that provides visual feedback to the user.

In certain embodiments, the processing and analysis of signals fromsensors embedded in electronic device 218 can detect when electronicdevice has been disabled, tampered with, removed from the body or is notbeing used. This can be used to conserve power, or to send anotification to the user, a friend or another person who might directlyor indirectly have an interest in being notified if electronic device218 is not being used properly.

Description Detection/Prediction of Start/End of Food Intake Event

In a preferred embodiment, the electronic device 218 is worn around thewrist, arm or finger and has one or more sensors that generate datanecessary to detect the start and/or end of a food intake event. Theelectronic device 218 may also be integrated in a patch that can beattached to a person's arm or wrist. The electronic device 218 may alsobe a module or add-on device that can be attached to another device thatis worn around the wrist, arm or finger. Sensors used to detect thestart and/or end of a food intake event may among other sensors includeone or more of the sensors described herein.

The raw sensor outputs may be stored locally in memory 629 and processedlocally on processing unit 628 to detect if the start or end of a foodintake event has occurred. Alternatively, one or more sensor outputs maybe sent to electronic device 219 and/or the central processing andstorage unit 220, either in raw or processed format, for furtherprocessing and to detect if the start or end of a food intake event hasoccurred. Regardless of where the processing for food intake detectionoccurs, sensor outputs in raw or processed format may be stored insideelectronic device 218, inside electronic device 219 and/or inside thecentral processing and storage unit 220.

The sensor or sensors that generate data necessary for the detection ofthe start and/or end of a food intake event may be internal toelectronic device 218. Alternatively, one or more of the sensorsresponsible for the detection of the start of a food intake event may beexternal to electronic device 218, but are able to relay relevantinformation to the electronic device 218 either directly through directwireless or wired, communication with electronic device 218 orindirectly, through another device. It is also possible that electronicdevice 218 and the external sensor or sensors area able to relayinformation to electronic device 219, but are not able to relayinformation to one another directly.

In case of indirect communication through another device such as amobile phone or other portable or stationary device, such third deviceis able to receive data or information from one or external sensorunits, optionally processes such data or information, and forwardseither the raw or processed data or information to electronic device218. The communication to and from the electronic device 218 may bewired or wireless, or a combination of both.

Examples of sensors that may be external to electronic device 218 may beone or more sensors embedded in a necklace or pendant worn around theneck, one or more sensors embedded in patches that are attached to adifferent location on the body, one or more sensors embedded in asupplemental second wearable device that is worn around the other arm orwrist or on a finger of the other hand, or one or more sensorsintegrated in a tooth. In some embodiments, the electronic device isworn on one hand or arm but detects movement of the other hand or arm.In some embodiments, electronic devices are worn on each hand.

Information obtained from the non-real-time analysis and learningsubsystem 105 may also be used, optionally in combination withinformation from one or more sensors 627, to predict or facilitate thedetection of a probable, imminent or actual start/end of a food intakeevent.

It is often desirable that the detection and/or the prediction of thestart and/or end of a food intake event happens autonomously withoutrequiring user intervention. For example, if the actual, probable orimminent start of a food intake event is predicted or detectedautonomously, this information can be used as a trigger to activate orpower up specific components or circuits that are only needed during afood intake event. This can help conserve power and extend the batterylife of electronic device 218. The prediction or detection of an actual,probable or imminent start of a food intake event can also be used toissue a cue or reminder to the user. A cue can for example be sent tothe user to remind him/her to take further actions including, but notlimited to, logging the food intake event or taking a picture of thefood. Upon detection of the start of a food intake event, one or morecues, possibly spread out over the duration of the food intake event, toremind the user that a food intake event is taking place and improvingin-the-moment awareness and/or encourage mindful eating. Cues orreminders may for example be sent through discrete haptic feedback usingone or more stimulus units 633. Other methods using one or more userinterfaces 632, such as for example one or more LEDs, a display message,or an audio signal, are also possible. Alternatively, mobile device 219may be used to communicate cues, reminders or other information such asfor example portion size recommendations or alternative suggestions toeating to the user.

If the actual, probable or imminent end of a food intake event ispredicted or detected autonomously, this information can be used as atrigger to power down or at least put in a lower power mode one or morecircuits or components of electronic device 218 that are only neededduring a food intake event. This can help conserve power and extend thebattery life of electronic device 218. The detection of the actual,probable or imminent end of a food intake event may also be used tomodify or suspend the feedback provided to the user by one or morestimulus units 633, by one or more of the user interfaces 632, and/or bymobile device 219.

In some embodiments of the present disclosure, the detection orprediction of the actual, probable or imminent start and/or end of afood intake event may not be entirely autonomously. For example, theuser may be required to make a specific arm, wrist, hand or fingergesture to signal to electronic device 218 the actual, probable orimminent start and/or end of a food intake event. The arm, wrist, handor finger gesture is then detected by one or more sensors insideelectronic device 218. It is usually desirable that the arm, wrist, handor finger gesture or gestures required to indicate the start and/or endof a food intake event can be performed in a subtle and discrete way.Other methods may also be used. For example, the user may be asked topush a button on electronic device 218 to indicate the start and/or endof a food intake event. Voice activation commands using a microphonethat is built into electronic device 18 may also be used. Other methodsare also possible.

Description of Tracking of Eating Behaviors and Patterns

In a particular embodiment, the electronic device 218 is worn around thewrist, arm or finger and has one or more sensors that generate data thatfacilitate the measurement and analysis of eating behaviors, patternsand habits. Sensors used for measuring and analyzing certain eatingbehaviors and patterns may include one or more of the sensors describedherein.

Relevant metrics that may be used to quantify and track eating behaviorsand eating patterns may include, but are not limited to, the timebetween subsequent bites or sips, the distance between the plate and theuser's mouth, the speed of arm movement towards and/or away from theuser's mouth, and the number of bites or sips during a single foodintake event, derived from the total count of arm movementscorresponding to a bite or sip, specific chewing behavior andcharacteristics, the time between taking a bite and swallowing, amountof chewing prior to swallowing.

The raw sensor outputs may be stored locally in memory 29 and processedlocally on processing unit 28. Alternatively, one or more sensor outputsmay be sent to electronic device 19 and/or the central processing andstorage unit 20, either in raw or in processed format, for furtherprocessing and analysis. Regardless of where the processing and analysisof eating behaviors and patterns occurs, sensor outputs in raw orprocessed format may be stored inside electronic device 18, insideelectronic device 19 and/or inside the central processing and storageunit 20.

In some embodiments, the generation, collection and/or processing ofdata that facilitate the measurement and analysis of eating behaviors,patterns and habits may be continuously, periodically or otherwiseindependently of the start and/or end of a food intake event.Alternatively, the generation, collection and/or processing of data thatfacilitate the measurement and analysis of eating behavior and patternsmay occur only during a food intake event or be otherwise linked to aspecific food intake event. It is also possible that some sensor dataare being generated, collected and/or processed continuously,periodically or otherwise independently of the start and/or end of afood intake event whereas other sensor data are taken during a foodintake event or otherwise linked to a food intake event.

The sensor or sensors that generate data necessary for measuring andanalyzing eating behaviors and eating patterns may be internal toelectronic device 18. Alternatively, one or more of the that generatedata necessary for measuring and analyzing eating behaviors and eatingpatterns may be external to electronic device 18, but are able to relayrelevant information to electronic device 18 either directly throughdirect wireless or wired, communication with electronic device 18 orindirectly, through another device.

In case of indirect communication through another device such as amobile phone or other portable or stationary device, such third deviceis able to receive data or information from the external sensor unit,optionally processes such data or information, and forwards either theraw or processed data or information to the tracking device. Thecommunication to and from the electronic device 18 may be wired orwireless, or a combination of both.

Examples of sensors that may be external to electronic device 18 may beone or more sensors embedded in a necklace or pendant worn around theneck, one or more sensors embedded in patches that are attached to adifferent location on the body, one or more sensors embedded in asupplemental second wearable device that is worn around the other arm orwrist or on a finger of the other hand, or one or more sensorsintegrated in a tooth.

Description of Use of Camera Module and Image Capture

While use of a camera to capture images of food have been proposed inthe prior art, they typically rely on the user taking pictures with hisor her mobile phone or tablet. Unfortunately, image capture using amobile phone or tablet imposes significant friction of use, may not besocially acceptable in certain dining situations or may interfere withthe authenticity of the dining experience. It is often times notdesirable or inappropriate that the user needs to pull out his or hermobile phone, unlock the screen, open a Mobile App and take a pictureusing the camera that is built into the mobile phone.

If user intervention is required, it is generally desirable that theuser intervention can be performed in a subtle and discrete manner andwith as little friction as possible. In order to minimize the frictionof use, it is often times desirable that the image capture can beinitiated from electronic device 18 directly.

While the examples provided herein use image capture of food and mealscenarios as examples, upon reading this disclosure, it should be clearthat the methods and apparatus described herein can be applied to imagecapture of objects and scenes other than foods and meal scenarios. Forexample, a viewfinder-less camera can have application outside of thefood event capture domain.

In some embodiments, electronic device 18 is worn around the wrist, armor finger and includes one or more camera modules 26. One or more cameramodules 26 may be used for the capture of still images in accordancewith one embodiment of the present disclosure, and for the capture ofone or more video streams in accordance with another embodiment of thepresent disclosure. In yet another embodiment of the present disclosure,a combination of still and streaming images is also possible.

One or more camera modules may also be included in a device that is wornat a different location around the body, such as a necklace or pendantthat is worn around the neck, or a device that is attached to orintegrated with the user's clothing, with the camera or camera modulespreferably aiming towards the front so that it can be in line of sightwith the food being consumed.

In some embodiments, activation of a camera module and/or image captureby a camera module may require some level of user intervention. Userintervention may, among other things, include pressing a button, issuinga voice command into a microphone that is built into electronic device18 or mobile device 19, making a selection using a display integrated inelectronic device 18 or mobile device 19, issuing a specific arm, wrist,hand or finger gesture, directing the camera so that the object ofinterest is within view of the camera, removing obstacles that may be inthe line of sight between the camera and the object of interest, and/oradjusting the position of the object of interest so that it is withinview of the camera. Other user intervention methods, or a combination ofmultiple user intervention methods are also possible.

In one embodiment of the present disclosure, a camera module is builtinto an electronic device, such as a wearable device, that may not havea viewfinder, or may not have a display that can give feedback to theuser about the area that is within view of the camera. In this case, theelectronic device may include a light source that projects a pattern ofvisible light onto a surface or onto an object to indicate to the userthe area that is within the view of the camera. One or more LightEmitting Diodes (LEDs) may be used as the light source. Other lightsources including, but not limited to, laser, halogen or incandescentlight sources are also possible. The pattern of visible light may, amongother things, be used by the user to adjust the position of the camera,adjust the position the object of interest and/or remove any objectsthat are obstructing the line of sight between the object of interestand the camera.

The light source may also be used to communicate other information tothe user. As an example, the electronic device may use inputs from oneor more proximity sensors, process those inputs to determine if thecamera is within the proper distance range from the object of interest,and use one or more light sources to communicate to the user that thecamera is within the proper distance range, that the user needs toincrease the distance between camera and the object of interest, or thatthe user needs to reduce the distance between the camera and the objectof interest.

The light source may also be used in combination with an ambient lightsensor to communicate to the user if the ambient light is insufficientor too strong for an adequate quality image capture.

The light source may also be used to communicate information including,but not limited, to a low battery situation or a functional defect.

The light source may also be used to communicate dietary coachinginformation. As an example, the light source might, among other things,indicate if not enough or too much time has expired since the previousfood intake event, or may communicate to the user how he/she is doingagainst specific dietary goals.

Signaling mechanisms to convey specific messages using one or more lightsources may include, but are not limited to, one or more of thefollowing: specific light intensities or light intensity patterns,specific light colors or light color patterns, specific spatial ortemporal light patterns. Multiple mechanisms may also be combined tosignal one specific message.

In another embodiment of the current disclosure, a camera module may bebuilt into an electronic device 18, such as a wearable device, that doesnot have a viewfinder or does not have a display that can give feedbackto the user about the area that is within view of the camera. Instead ofor in addition to using a light source, one or more images captured bythe camera module, possibly combined with inputs from other sensors thatare embedded in electronic device 18 may be sent to the processing unitinside electronic device 18, the processing unit inside electronicdevice 19, and/or the central processing and storage unit 20 foranalysis and to determine if

If the object of interest is within proper view and/or proper focalrange of the camera. The results of the analysis may be communicated tothe user using one of the feedback mechanisms available in electronicdevice 18 including, but not limited to, haptic feedback, visualfeedback using one or more LEDs or a display, and/or audio feedback.

In some other embodiments of the present disclosure, electronic device18 may capture one or more images without any user intervention.Electronic device 18 may continuously, periodically or otherwiseindependently of any food intake event capture still or streamingimages. Alternatively, electronic device 18 may only activate one ormore of its camera modules around or during the time of a food intakeevent. As an example, an electronic device may only activate one or moreof its camera modules and capture one or more images after the start ofa food intake event has been detected and before the end of a foodintake event has been detected. It may use one or more of its cameramodules to capture one of more images of food items or dishes in theirentirety, or of a portion of one or more food items or dishes.

In some embodiments, one camera may be used to capture one or moreimages of food items that are on a plate, table or other stationarysurface, and a second camera may be used to capture one or more imagesof food items that are being held by the user, such as for examplefinger foods or drinks. The use of more than one camera may be desirablein situations where no user intervention is desirable and the position,area of view or focal range of a single camera is not suite to captureall possible meal scenarios.

In one example embodiment, the position, the orientation and the angleof view of the camera are such that an image or video capture ispossible without any user intervention. In such an embodiment, thewearable device may use a variety of techniques to determine the propertiming of the image or video stream capture such that it can capture thefood or a portion of the food being consumed. It may also choose tocapture multiple images or video streams for this purpose. Techniques todetermine the proper timing may include, but are not limited to, thefollowing: sensing of proximity, sensing of acceleration or motion (orabsence thereof), location information. Such sensor information may beused by itself or in combination with pattern recognition or dataanalytics techniques (or a combination of both) to predict the besttiming for the image or video capture. Techniques may include, but arenot limited to, training of a model based on machine learning.

The captured still and/or streaming images usually require some level ofprocessing. Processing may include but is not limited to compression,deletion, resizing, filtering, image editing, and computer visiontechniques to identify objects such as for example specific foods ordishes, or features such as for example portion sizes. Processing unitsthat may be used to process still or streaming images from the cameramodule or modules, regardless of whether or not the camera module ormodules are internal to the electronic device 18, include, but are notlimited to, the processing unit inside the electronic device 18, theprocessing unit inside electronic device 19 and/or a central processingand storage unit 20 which may reside at the same location as where theelectronic device is being used or alternatively, may reside at a remotelocation (e.g., in a cloud server) in which case it may be accessed viathe internet. The image processing may also be distributed among acombination of the abovementioned processing units.

Examples of local processing may include but are not limited to:selection of one or more still images out of multiple images or one ormore video streams, compression of images or video stream, applicationof computer vision algorithms on one or more images or video streams.

Local processing may include compression. In case of compression, acompressed image may be transmitted as part of a time criticaltransaction whereas its non-compressed version may be saved fortransmission at a later time.

One or more still or streaming images may be analyzed and/or comparedfor one or multiple purposes including, but not limited to, thedetection of the start and/or end of a food intake event, theidentification of food items, the identification of food content, theidentification or derivation of nutritional information, the estimationof portion sizes and the inference of certain eating behaviors andeating patterns.

As one example, computer vision techniques, optionally combined withother image manipulation techniques may be used to identify foodcategories, specific food items and/or estimate portion sizes.Alternatively, images may be analyzed manually using a Mechanical Turkprocess or other crowdsourcing methods. Once the food categories and/orspecific food items have been identified, this information can be usedto retrieve nutritional information from one or more foods/nutritiondatabases.

As another example, information about a user's pace of eating ordrinking may be inferred from analyzing and comparing multiple imagescaptured at different times during the course of a food intake event. Asyet another example, images, optionally combined with other sensorinformation, may be used to distinguish a sit-down meal from fingerfoods or snacks. As yet another example, the analysis of one image takenat the start of a food intake event and another image taken at the endof a food intake event may provide information on the amount of foodthat was actually consumed.

Description of User Feedback

In a preferred embodiment of the present disclosure, the electronicdevice 18 is worn around the wrist, arm or finger and has one or morestimulus units and/or user interfaces that allow for feedback to theuser or the wearer of the electronic device. In a different embodimentof the present disclosure, electronic device 18 may be implemented as awearable patch that may be attached to the body or may be embedded inclothing.

Feedback usually includes feedback that is food or food intake related.Feedback methods may include, but are not limited to, haptic feedback,visual feedback using LEDs or a display or audio feedback. In one suchembodiment, electronic device 18 may have a haptic interface thatvibrates once or multiple times when the start and/or end of a foodintake event have been detected. In another embodiment, electronicdevice 18 may have a haptic interface that vibrates once or multipletimes when the tracking and processing subsystem identifies that thewearer of the device is consuming food and is showing eating behaviorthat is exceeding certain programmed thresholds, such as for exampleeating too fast, too slow or too much. Alternatively, the hapticinterface may vibrate one or more times during a food intake event,independent of any specific eating behavior, for example to remind thewearer of the fact that a food intake event is taking place and/or toimprove in-the-moment awareness and to encourage mindful eating. Otherfeedback methods are also possible, and different metrics or criteriamay be used to trigger an activation of such feedback methods.

In a different embodiment of the present disclosure, feedback isprovided to the user through a device that is separate from theelectronic device 18. One or more stimulus units and/or user interfacesrequired to provide feedback to the user may be external to electronicdevice 18. As an example, one or more stimulus units and/or userinterfaces may be inside electronic device 19, and one or more of saidstimulus units and/or user interfaces inside electronic device 19 may beused to provide feedback instead of or in addition to feedback providedby electronic device 18. Examples may include, but are not limited to,messages being shown on the display of electronic device 19, or soundalarms being issued by the audio subsystem embedded inside electronicdevice 19.

Alternatively, feedback may be provided through a device that isseparate from both electronic device 18 and electronic device 19, butthat is able to at a minimum, either directly or indirectly, receivedata from at least one of those devices.

In addition to or instead of feedback provided around or during the timeof a food intake event, the system of FIG. 2 or FIG. 3 may also providefeedback that may span multiple food intake events or may not linked toa specific food intake event or set of food intake events. Examples ofsuch feedback may include, but are not limited to, food content andnutritional information, historical data summaries, overviews of one ormore tracked parameters over an extended period of time, progress of oneor more tracked parameters, personalized dietary coaching and advice,benchmarking of one or more tracked parameters against peers or otherusers with similar profile.

DETAILED DESCRIPTION OF SPECIFIC EMBODIMENTS

In one specific embodiment of the present disclosure, electronic device218 is a wearable device in the form factor of a bracelet or wristbandthat is worn around the wrist or arm of a user's dominant hand.Electronic device 219 is a mobile phone and central processing andstorage unit 220 is one or more compute servers and data storage thatare located at a remote location.

One possible implementation of a wearable bracelet or wristband inaccordance with aspects of the present invention is shown in FIG. 7.Wearable device 770 may optionally be implemented using a modulardesign, wherein individual modules include one or more subsets of thecomponents and overall functionality. The user may choose to addspecific modules based on his personal preferences and requirements.

The wearable device 770 may include a processor, a program code memoryand program code (software) stored therein and/or inside electronicdevice 219 to optionally allow users to customize a subset of thefunctionality of wearable device 770.

Wearable device 770 relies on battery 769 and Power Management Unit(“PMU”) 760 to deliver power at the proper supply voltage levels to allelectronic circuits and components. Power Management Unit 760 may alsoinclude battery-recharging circuitry. Power Management Unit 760 may alsoinclude hardware such as switches that allows power to specificelectronics circuits and components to be cut off when not in use.

When there is no behavior event in progress, most circuitry andcomponents in wearable device 770 are switched off to conserve power.Only circuitry and components that are required to detect or helppredict the start of a behavior event may remain enabled. For example,if no motion is being detected, all sensor circuits but theaccelerometer may be switched off and the accelerometer may be put in alow-power wake-on-motion mode or in another lower power mode thatconsumes less power than its high performance active mode. Theprocessing unit may also be placed into a low-power mode to conservepower. When motion or a certain motion pattern is detected, theaccelerometer and/or processing unit may switch into a higher power modeand additional sensors such as for example the gyroscope and/orproximity sensor may also be enabled. When a potential start of an eventis detected, memory variables for storing event-specific parameters,such as gesture types, gesture duration, etc. can be initialized.

In another example, upon detection of motion, the accelerometer switchesinto a higher power mode, but other sensors remain switched off untilthe data from the accelerometer indicates that the start of a behaviorevent has likely occurred. At that point in time, additional sensorssuch as the gyroscope and the proximity sensor may be enabled.

In another example, when there is no behavior event in progress, boththe accelerometer and gyroscope are enabled but at least one of eitherthe accelerometer or gyroscope is placed in a lower power mode comparedto their regular power mode. For example, the sampling rate may bereduced to conserve power. Similarly, the circuitry required to transferdata from electronic device 218 to electronic device 219 may be placedin a lower power mode. For example, radio circuitry 764 could bedisabled completely. Similarly, the circuitry required to transfer thedata from electronic device 218 to electronic device 219 may be placedin a lower power mode. For example, it could be disabled completelyuntil a possible or likely start of a behavior event has beendetermined. Alternatively, it may remain enabled but in a low powerstate to maintain the connection between electronic device 218 andelectronic device 219 but without transferring sensor data.

In yet another example, all motion-detection related circuitry,including the accelerometer may be switched off, if based on certainmeta-data it is determined that the occurrence of a particular behaviorevent such as a food intake event is unlikely. This may for example bedesirable to further conserve power. Meta-data used to make thisdetermination may, among other things, include one or more of thefollowing: time of the day, location, ambient light levels, proximitysensing, and detection that wearable device 770 has been removed fromthe wrist or hand, detection that wearable device 770 is being charged.Meta-data may be generated and collected inside wearable device 770.Alternatively, meta-data may be collected inside the mobile phone orinside another device that is external to wearable device 770 and to themobile phone and that can either directly or indirectly exchangeinformation with the mobile phone and/or wearable device 770. It is alsopossible that some of the meta-data are generated and collected insidewearable device 770 whereas other meta-data are generated and collectedin a device that is external to wearable device 770. In case some or allof the meta-data are generated and collected external to wearable device770, wearable device 770 may periodically or from time to time power upits radio circuitry 764 to retrieve meta-data related information fromthe mobile phone or other external device.

In yet another embodiment of the invention, some or all of the sensorsmay be turned on or placed in a higher power mode if certain meta-dataindicates that the occurrence of a particular behavior event, like forexample a food intake event is likely. Meta-data used to make thisdetermination may, among other things, include one or more of thefollowing: time of the day, location, ambient light levels and proximitysensing. Some or all of the meta-data may be collected inside the mobilephone or inside another device that is external to wearable device 770and to the mobile phone and that can either directly or indirectlyexchange information with the mobile phone and/or wearable device 770.In case some or all of the meta-data are generated and collectedexternal to wearable device 770, wearable device 770 may periodically orfrom time to time power up its radio circuitry 764 to retrieve meta-datarelated information from the mobile phone or other external device.

The detection of the start of a behavior event, such as for example afood intake event may be signaled to the user via one of the availableuser interfaces on wearable device 770 or on the mobile phone to whichwearable device 770 is connected. As one example, haptic interface 761inside wearable device 770 may be used for this purpose. Other signalingmethods are also possible.

The detection of the start of a behavior event such as for example afood intake event may trigger some or all of the sensors to be placed orremain in a high-power mode or active mode to track certain aspects of auser's eating behavior for a portion or for the entirety of the foodintake event. One or more sensors may be powered down or placed in alower power mode when or sometime after the actual or probable end ofthe behavior event (the deemed end of the behavior event) has beendetected. Alternatively, it is also possible that one or more sensorsare powered down or placed in a lower power mode after a fixed orprogrammable period of time.

Sensor data used to track certain aspects of a user's behavior, such asfor example a user's eating behavior, may be stored locally insidememory 766 of wearable device 770 and processed locally using processingunit 767 inside wearable device 770. Sensor data may also be transferredto the mobile phone or remote compute server, using radio circuitry 764,for further processing and analysis. It is also possible that some ofthe processing and analysis is done locally inside wearable device 770and other processing and analysis is done on the mobile phone or on aremote compute server.

The detection of the start of a behavior event, such as for example thestart of a food intake event, may trigger the power up and/or activationof additional sensors and circuitry such as for example the cameramodule 751. Power up and/or activation of additional sensors andcircuitry may happen at the same time as the detection of the start of afood intake event or sometime later. Specific sensors and circuitry maybe turned on only at specific times during a food intake event whenneeded and may be switched off otherwise to conserve power.

It is also possible that the camera module only gets powered up oractivated upon explicit user intervention such as for example pushingand holding a button 759. Releasing the button may turn off the cameramodule again to conserve power.

When the camera module 751 is powered up, projecting light source 752may also be enabled to provide visual feedback to the user about thearea that is within view of the camera. Alternatively, projecting lightsource 752 may only be activated sometime after the camera module hasbeen activated. In certain cases, additional conditions may need to bemet before projecting light source 752 gets activated. Such conditionsmay, among other things, include the determination that projecting lightsource 752 is likely aiming in the direction of the object of interest,or the determination that wearable device 752 is not moving excessively.

In one specific implementation, partially depressing button 759 onwearable device 770 may power up the camera module 751 and projectinglight source 752. Further depressing button 759 may trigger cameramodule 751 to take one or more still images or one or more streamingimages. In certain cases, further depressing button 759 may trigger ade-activation, a modified brightness, a modified color, or a modifiedpattern of projected light source 752 either before or coinciding withthe image capture. Release of button 759 may trigger a de-activationand/or power down of projected light source 752 and/or of camera module751.

Images may be tagged with additional information or meta-data such asfor example camera focal information, proximity information fromproximity sensor 756, ambient light levels information from ambientlight sensor 757, timing information etc. Such additional information ormeta-data may be used during the processing and analysis of food intakedata.

Various light patterns are possible and may be formed in various ways.For example, it may include a mirror or mechanism to reflect projectinglight source 752 such that projected light source 752 produces one ormore lines of light, outlines the center or boundaries a specific area,such as a cross, L-shape, circle, rectangle, multiple dots or linesframing the field of view or otherwise giving to the user visualfeedback about the field of view.

One or more Light Emitting Diodes (LEDs) may be used as project lightsource 752. The pattern of visible light may, among other things, beused by the user to adjust the position of the camera, adjust theposition the object of interest and/or remove any objects that areobstructing the line of sight between the object of interest and thecamera.

Projected light source 752 may also be used to communicate otherinformation to the user. As an example, the electronic device me useinputs from one or more proximity sensors, process those inputs todetermine if the camera is within the proper distance range from theobject of interest, and use one or more light sources to communicate tothe user that the camera is within the proper distance range, that theuser needs to increase the distance between camera and the object ofinterest, or that the user needs to reduce the distance between thecamera and the object of interest.

The light source may also be used in combination with an ambient lightsensor to communicate to the user if the ambient light is insufficientor too strong for an adequate quality image capture.

The light source may also be used to communicate information including,but not limited to, a low battery situation or a functional defect.

The light source may also be used to communicate dietary coachinginformation. As an example, the light source might, among other things,indicate if not enough or too much time has expired since the previousfood intake event, or may communicate to the user how he/she is doingagainst specific dietary goals.

Signaling mechanisms to convey specific messages using one or moreprojecting light sources may include, but are not limited to, one ormore of the following: specific light intensities or light intensitypatterns, specific light colors or light color patterns, specificspatial or temporal light patterns. Multiple mechanisms may also becombined to signal one specific message.

Microphone 758 may be used by the user to add specific or custom labelsor messages to a food intake event and/or image. Audio snippets may beprocessed by a voice recognition engine.

In certain embodiments, the accelerometer possibly combined with othersensors may, in addition to tracking at least one parameter that isdirectly related to food intake and/or eating behavior, also be used totrack one or more parameters that are not directly related to foodintake. Such parameters may, among other things, include activity, sleepor stress.

Specific Embodiments without Built-in Camera

In a different embodiment, electronic device 218 may not have anybuilt-in any image capture capabilities. Electronic device 218 may be awearable device such as a bracelet or wristband worn around the arm orwrist, or a ring worn around the finger. Electronic device 219 may be amobile phone and central processing and storage unit 220 may be one ormore compute servers and data storage that are located at a remotelocation.

In such embodiments, the food intake tracking and feedback system maynot use images to extract information about food intake and/or eatingbehavior. Alternatively, the food intake tracking and feedback systemmay leverage image capture capabilities that are available inside otherdevices, such as for example electronic device 219 or otherwise anelectronic device that is external to electronic device 218.

Upon detection or prediction of the start of a food intake event,electronic device 218 may send a signal to electronic device 219, or tothe electronic device that is otherwise housing the image capturecapabilities to indicate that the actual, probable or imminent start ofa food intake event has occurred. This may trigger electronic device219, or the electronic device that is otherwise housing the imagecapture capabilities to enter a mode that will allow the user to capturean image with at least one less user step compared to its default modeor standby mode.

As an example, if the image capture capabilities are housed withinelectronic device 219 and electronic device 219 is a mobile phone, atablet or a similar mobile device, electronic device 218 may send one ormore signals to software that has been installed on electronic device219 to indicate the actual, probable or imminent start of a food intakeevent. Upon receiving such signal or signals, the software on electronicdevice 219 may, among other things, take one or more of the followingactions: unlock the screen of electronic device 219, open the MobileApplication related to the food intake and feedback subsystem, activateelectronic device's 219 camera mode, push a notification to electronicdevice's 219 display to help a user with image capture, send a messageto electronic device 218 to alert, remind and/or help a user with imagecapture.

After image capture by electronic device 219, or the electronic devicethat is otherwise housing the image capture capabilities, electronicdevice 219, or the electronic device that is otherwise housing the imagecapture capabilities, may give visual feedback to the user. Examples ofvisual feedback may include a pattern, shape or overlay showingrecommended portion sizes, or a pattern, shape or overlay shade in oneor more colors and/or with one or more brightness levels to indicate howhealthy the food. Other examples are also possible.

Integration with Insulin Therapy System

One or more components of the food intake tracking and feedback systempresented in this disclosure may interface to or be integrated with aninsulin therapy system. In one specific example, upon detection of thestart of a food intake event, feedback may be sent to the wearer toremind him or her to take a glucose level measurement and/or administerthe proper dosage of insulin. One or more additional reminders may besent over the course of the food intake event.

The food intake tracking and feedback system described in thisdisclosure, or components thereof may also be used by patients who havebeen diagnosed with Type I or Type II diabetes. For example, componentsdescribed in the current disclosure may be used to detect automaticallywhen a person starts eating or drinking. The detection of the start of afood intake event may be used to send a message to the wearer at or nearthe start of a food intake event to remind him or her to take a glucoselevel measurement and/or administer the proper dosage of insulin. Themessaging may be automatic and stand alone. Alternatively, the systemmay be integrated with a wellness system or a healthcare maintenance andreminder system. The wellness system or the healthcare maintenance andreminder system may upon getting notified that the start of a foodintake event has been detected send a message to the wearer. Thewellness system or the healthcare maintenance and reminder system mayreceive additional information about the food intake event, such as thenumber of bites or sips, the estimated amount of food consumed, theduration of the meal, the pace of eating etc. The wellness system or thehealthcare maintenance and reminder system may send additional messagesto the wearer during or after the food intake event based on theadditional information.

In another example, specific information about the content of the foodintake may be used as an input, possibly combined with one or more otherinputs, to compute the proper dosage of insulin to be administered.Information about food intake content may, among other things, includeone or more of the following: amount of carbohydrates, amounts ofsugars, amounts of fat, portion size, and molecular food category suchas solids or liquids. Real-time, near real-time as well as historicalinformation related food intake and eating patterns and behaviors may beincluded as inputs or parameters for computation of insulin dosages.

Other inputs that may be used as inputs or parameters to the algorithmsthat are used to compute insulin dosages may include, among otherthings, one or more of the following: age, gender, weight, historicaland real-time blood glucose levels, historical and real-time activity,sleep and stress levels, vital sign information or other informationindicative of the physical or emotional health of an individual.

Computation of insulin dosages may be done fully manually by the user,fully autonomously by a closed loop insulin therapy system orsemi-autonomously where some or all of the computation is done by aninsulin therapy system but some user intervention is still required.User intervention may, among other things, include activation of theinsulin therapy computation unit, confirmation of the dosage, interveneor suspend insulin delivery in case user detects or identifies ananomaly.

In one specific embodiment, the food intake tracking and feedback systemdescribed herein may upon detection of the actual, probable or imminentstart of a food intake event send one or more notifications to one ormore caregivers of the user, in addition or instead of sending anotification to the user.

The user may upon the start of a food intake event, optionally promptedby a notification or signal from the system or from one his caregiver,take one or more images of the food or meal to one or more caregiver.The caregiver may analyze the images and send information about thecontent of the food back to the user. Information may, among otherthings, include estimation of certain macro-nutrient contents such asfor example carbohydrates, sugars or fats, estimation of caloric value,advice on portion size.

In case the user is on an insulin therapy, additional information suchas for example blood glucose level readings may also be sent to thecaregiver, and information provided by a caregiver back to a user mayalso include advice on the insulin dosage to be administered and thetiming when such insulin dosage or dosages should be administered. Incertain implementations, the caregiver may not be a person but anartificial intelligence system.

Gesture Recognition

In the various systems described herein, accurate determination ofgesture information can be important. For example, it would be useful todistinguish between a gesture connected with talking versus a gesturethat signals the start of an eating event period. Some gestures might beeasy to detect, such as the gesture of swinging an arm while walking,and thus measuring pace and number of steps, but other gestures might bemore difficult, such as determining when a user is taking a bite offood, taking a drink, biting their nails, etc. The latter can be usefulfor assessing precursor behaviors. For example, suppose a healthmaintenance and reminder system detects a pattern of nail bitinggestures followed five to ten minutes later with gestures associatedwith stress eating. The user might program their health maintenance andreminder system to signal them two minutes after nail biting so that theuser becomes aware and more in tune with their behavior that wouldotherwise go unnoticed. For this to work, gesture detection should beaccurate and reliable. This can be a problem where there is not a simplecorrelation between, say, movement of an accelerometer in a wearablebracelet and stress eating. Part of the reason for this is that thegestures that are of interest to the health maintenance and remindersystem are not easily derived from a simple sensor reading.

Being able to determine whether a user is taking a bite of food ortaking a sip of a drink, and being able to distinguish a bite from asip, can be useful to provide proper weight management guidance. Forexample, a weight management monitoring and reminder system may monitora user's food intake events from gestures. The weight managementmonitoring a reminder system may furthermore monitor a user's fluidintake events from gestures. Studies have shown that drinking sufficientwater at the start or close to the start of a meal and further drinkingsufficiently throughout the meal reduces food consumption and helps withweight loss. The user, the user's coach, the user's healthcare provider,or the provider of the weight management monitoring and reminder systemmay program the system such that it sends a reminder when a user startseating without drinking or if it detects that the user is not drinkingsufficiently throughout the meal. The system could also monitor theuser's fluid intake throughout the day and be programmed to sendreminders if the level of fluid intake does not meet the pre-configuredlevel for a particular time of day. For this to work, the gesturedetection should be reliable and accurate. This can be a problem whereit is necessary to distinguish between gestures that have lots ofsimilarities, such as for example distinguishing an eating gesture froma drinking gesture.

In various embodiments described herein, a processing system (comprisingprogram code, logic, hardware, and/or software, etc.) takes in sensordata generated by electronic devices or other elements based onactivities of a user. The sensor data might represent a snapshot of areading at a specific time or might represent readings over a span oftime. The sensors might be accelerometers, gyroscopes, magnetometers,thermometers, light meters and the like. From the sensor data, theprocessing system uses stored rules and internal data (such asinformation about what sensors are used and past history of use) toidentify behavior events wherein a behavior event is a sequence ofgestures and the gestures are determined from logical arrangement ofsensor data having a start time, sensor readings, and an end time, aswell as external data. The behavior event might be a high-level event,such as eating a meal, etc.

The determination of the boundaries of gestures, i.e., their start andend times, can be determined using methods described herein. Together,the data of a start time, sensor readings, and an end time is referredto herein as a gesture envelope. The gesture envelope might also includean anchor time, which is a data element defining a single time that isassociated with that gesture envelope. The anchor time might be themidpoint between the start time and the end time, but might be based onsome criteria based on the sensor data of the gesture envelope. Ananchor time might be outside of the time span from the start time to theend time. Multiple anchor times per gesture are also possible.

A machine classifier, as part of the processing system (but can also bea separate computer system, and possibly separated by a network of somekind), determines from a gesture envelope what class of gesture mighthave resulted in that gesture envelope's sensor data and details of thegesture. For example, the machine classifier might output that thesensor data indicates or suggests that a person wearing a bracelet thatincludes sensors was taking a walk, talking a bite to eat, or pointingat something.

With such a system, if gestures can be accurately discerned, then ahealth maintenance and reminder system (or other system that usesgesture information) can accurately respond to gestures made. In anexample described below, there is a set of sensors, or at least inputsfrom a set of sensors, coupled to a machine classification system thatoutputs gesture data from sensor readings, taking into account rules andstored data derived from training the machine classification system. Atraining subsystem might be used to train the machine classificationsystem and thereby forming the stored data derived from training. Eachof these components might use distinct hardware, or shared hardware, andcan be localized and/or remote. In general, when a gesture is detected,a system can analyze that gesture, determine likely actual, probable orimminent activities and provide the user feedback with respect to thoseactivities. For example, a vibration as a feedback signal to indicatethat the user has previously set up the system to alert the user whenthe user has been drinking for a semi-continuous period of more than 45minutes or that the user has reached their target for the amount ofwalking to be done in one session.

FIG. 8 is an illustrative example of a typical machine classificationsystem. The machine classification system of FIG. 8 includes a trainingsubsystem 801 and a detector subsystem 802. In some embodiments of thepresent disclosure, the machine classification system may includeadditional subsystems or modified versions of the subsystems shown inFIG. 8. Training subsystem 801 uses training data inputs 803 and labels804 to train trained classifier model 805. Labels 804 may have beenassigned manually by a human or may have been generated automatically orsemi-automatically. Trained classifier model 805 is then used indetector subsystem 802 to generate classification output 806corresponding to a new unlabeled data input.

The stored sensor data includes temporal components. Raw sensor readingsare tagged with their time of reading. The raw sensor data can be drawnfrom accelerometers, gyroscopes, magnetometers, thermometers,barometers, humidity sensors, ECG sensors and the like, and temporaldata can come from other sources. Other examples of temporal sourcesmight be audio, voice or video recordings.

Illustrative examples of training subsystem 801 and detector subsystem802 in accordance with at least one embodiment of the present disclosureare shown in FIG. 9 and FIG. 10 respectively. Temporal training data 907and labels 912 are fed into classifier training subsystem of FIG. 8.

As explained in the examples herein, raw sensor data is processed toidentify macro signature events. The macro signature events can delimitgestures that comprise sensor data over a period of time. The detectorsubsystem, or other system, can create a gesture envelope datasetcomprising a start time, an end time, one or more anchor times, metadataand sensor data that occurred within that gesture's time envelope fromthe start time to the end time.

For example, in the case of a gesture recognition problem, the gestureenvelope detector may identify specific time segments in the rawtemporal data that are indicative of a possible gesture. The gestureenvelope detector also generates a time envelope that specifies relevanttimes or segments of time within the gesture. Information included inthe time envelope may among other things include start time of thegesture, end time of the gesture, time or times within the gesture thatspecify relevant gesture sub-segments, time or times within the gesturethat specify relevant gesture anchor times (points) and possibly othermetadata, and raw sensor data from within the gesture's time envelope.

As an example of other metadata, suppose historical patterns suggestthat a wearer would have an eating session following a telephone callfrom a particular phone number. The electronic device can signal to thewearer about this condition, to provide conscious awareness of thepattern, which can help alter behavior if the wearer so decides.

Temporal training data 907 are fed into an gesture envelope detector908. Gesture envelope detector 908 processes temporal training data 907and identifies possible instances of gestures 909 and a correspondinggesturetime envelope from temporal training data 907. Temporal trainingdata 907 may comprise motion sensor data and gesture envelope detector908 may be processing the motion sensor data and identify gestures 909based on changes in pitch angle. In one embodiment, gesture envelopedetector 908 may detect the start of a gesture based on the detection ofa rise in pitch angle above a specified value and the end of an eventbased on the pitch angle dropping below a specified value. Other startand end criteria are also possible. An example of anchor points that maybe detected by gesture envelope detector 908 and specified by thegesture time envelope would be the time within the gesture segment whenthe pitch angle reaches a maximum. Other examples of anchor points arealso possible.

Gesture envelope detector 908 may add additional criteria to furtherqualify the segment as a valid gesture. For example, a threshold couldbe specified for the peak pitch angle or the average pitch angle withinthe segment. In another example, minimum and/or maximum limits may bespecified for the overall segment duration or for the duration ofsub-segments within the overall segment. Other criteria are alsopossible. Hysteresis may be employed to reduce the sensitivity to noisejitters.

In other embodiments of the present disclosure, gesture envelopedetector 908 may monitor other metrics derived from the input providingtemporal training data 907 and use those metrics to detect gestures.Examples of other metrics include but are not limited to roll angle,yaw, first or higher order derivative, or first or higher orderintegration of motion sensor data. Temporal data may be or may include,data other than motion sensor data. In some embodiments of the presentdisclosure, an gesture envelope detector 908 may monitor and usemultiple metrics to detect gestures or to specify the gesture timeenvelope.

Gestures 909 along with gesture time envelope information, combined withtemporal training data 907 are fed into a feature generator module 910.Feature generator module 910 computes one or more gesture features usinginformation from temporal training data 907, the gesture time envelope,or a combination of information from temporal training data 907 and thegesture time envelope. In some embodiments of the present disclosure,feature generator module 910 computes one or more gesture features fromtemporal training data 907 within or over a time segment that fallswithin the gesture time envelope. It is also possible that featuregenerator module 910 computes one or more gesture features from temporaltraining data 907 within or over a time segment that does not fallwithin or that only partially falls within the gesture time envelope,but that is still related to the gesture time envelope. An example wouldbe an gesture feature that is computed from temporal training data 907over a time period immediately preceding the start of the gesture timeenvelope or over a time period immediately following the end of thegesture time envelope.

In some embodiments, feature generator module 910 may create one or morefeatures based on gesture time envelope information directly withoutusing temporal training data 907. Examples of such features may include,but are not limited to, total duration of the gesture time envelope,elapsed time since a last prior gesture, a time until next gesture, ordurations of specific sub-segments within the overall gesture timeenvelope or event time envelope.

In one embodiment, temporal training data 907 may be motion sensor dataand features may include read of pitch, roll and/or yaw angles computedwithin, or over, one or more sub-segments that are inside or around thegesture time envelope. Features may also include minimum, maximum, mean,variance, first or higher order derivative, first or higher orderintegrals of various motion sensor data inputs computed within or overone or more sub-segments that are inside or around the gesture timeenvelope. Features may also include distance traveled along a specificsensor axis or in a specific direction computed within or over one ormore sub-segments that are inside or around the gesture time envelope.Other features are also possible.

Temporal training data 907 may be, or may include, data other thanmotion sensor data, such as sensor signals from one or more of thesensors described herein. Sub-segments within or over which featuregenerator module 910 computes features may be chosen based on timepoints or time segments specified by the gesture time envelope.Sub-segments may also be chosen based on time points or time segmentsfrom multiple gesture envelopes, such as for example adjacent gesturesor gestures that are may not be adjacent but are otherwise in closeproximity.

Some embodiments may use a plurality of gesture envelope detectors, inparallel or otherwise. Parallel gesture envelope detectors may operateon a different subset of the sensor data, may use different thresholdsor criteria to qualify gestures, etc. For example, in case of gesturerecognition based on motion sensor data inputs, one gesture envelopedetector may use the pitch angle, whereas a second, parallel gestureenvelope detector may use roll angle. One of the gesture envelopedetectors may be the primary gesture envelope detector, whereas one ormore additional gesture envelope detectors may serve as secondarygesture envelope detectors. The Feature Generation logic may processgestures generated by the primary gesture envelope detector, but maygleam features derived using information from gesture time envelopes ofnearby gestures (in time) obtained from one or more secondary, parallelenvelope detectors.

Training data might comprise a plurality of gesture envelope datasets,each having an associated label representing a gesture (such as aselection from a list of gesture labels), provided manually, in a testenvironment, or in some other manner. This training data, with theassociated labels, can be used to train the machine classifier, so thatit can later process an gesture envelope of an unknown gesture anddetermine the gesture label most appropriately matching that gestureenvelope. Depending on the classification method used, the training setmay either be cleaned, but otherwise raw data (unsupervisedclassification) or a set of features derived from cleaned, but otherwiseraw data (supervised classification).

Regardless of the classification method, defining the proper databoundaries for each label is important to the performance of theclassifier. Defining the proper data boundaries can be a challenge incase of temporal problems, i.e., problems whereby at least one of thedata inputs has a time dimension associated with it. This isparticularly true if the time dimension is variable or dynamic and iffeatures that are linked to specific segments of the variable timeenvelope or to the overall variable time envelope contribute materiallyto the performance of the classifier.

One example of such a temporal problem is gesture recognition, such asfor example the detection of an eating or drinking gesture from rawmotion sensor data. The duration of a bite or sip may varyperson-to-person and may depend on the meal scenario or specifics of thefoods being consumed. Examples of other gesture recognition problems arerecognition of hand gestures related to smoking, dental hygiene, nailbiting, nose picking, hair pulling, sign language, etc. In somevariations, the system is used in a production environment to improveproductivity.

The Feature Generation logic may also create features derived fromcombining outputs from multiple gesture envelope detector outputs.Examples include but are not limited to the elapsed time from a primarygesture to the nearest gesture from a parallel, secondary gestureenvelope detector.

The output of the feature generator module 910 is a set of gestures 911with corresponding time envelope and features. Before gestures 911 canbe fed into Classifier Training module 915, labels 912 from the trainingdataset need to be mapped to their corresponding gesture. This mappingoperation is performed by the Label Mapper module 913.

In some embodiments, the timestamps associated with labels 912 alwaysfall within the time envelope of their corresponding gesture. In thatcase, the logic of Label Mapper module 913 can be a look up where thetimestamp of each label is compared to the start and end time of eachgesture time envelope and each label is mapped to the gesture for whichthe timestamp of the label is larger than the start time of therespective gesture time envelope and smaller than the end time of therespective gesture time envelope. Gestures for which there is nocorresponding label may be labeled as “NEGATIVE”, indicating it does notcorrespond to any labels of interest.

However, in other embodiments of the present disclosure, the timestampof labels 912 may not always fall within an gesture time envelope. Thismay be due to the specifics of the procedures that were followed duringthe labeling process, timing uncertainty associated with the labelingprocess, unpredictability or variability in the actual raw data input,or an artifact of the gesture envelope detector logic. In such cases,the label mapper might be modified to adjust the boundaries of thegesture envelopes.

Gestures 914, characterized by features and a label, may then be fedinto Classifier Training module 915 to produce a trained statisticalmodel that can be used by the Detector subsystem. Classifier Trainingmodule 915 may use a statistical model such as a decision tree model, aK-nearest neighbors model, a Support Vector Machine model, a neuralnetworks model, a logistic regression model or other model appropriatefor a machine classification. In other variations, the structures of thetables and the data formats of the data used, as in FIG. 9, may vary andbe different from that shown in FIG. 9.

FIG. 10 shows an illustrative example of a detector subsystem. As shownthere, unlabeled temporal data 1017 is fed into the detector subsystemof FIG. 10. The detector subsystem includes gesture envelope detectorlogic 1018 and feature generator logic 1020. Functionally, gestureenvelope detector logic 1018 used by the detector subsystem is similarto gesture envelope detector logic used by its corresponding trainingsubsystem. Likewise, feature generator logic 1020 of the detectorsubsystem is functionally similar to feature generator module 910 of itscorresponding training subsystem. In some embodiments, gesture envelopedetector logic 1018 may monitor and use multiple metrics to detectgestures or to specify the gesture time envelope.

However, the implementation of gesture envelope detector logic 1018 andfeature generator logic 1020 may be different in the training subsystemand its corresponding detector subsystem. For example, the detectorsubsystem may be implemented on hardware that is more power-constrained,in which case gesture envelope detector logic 1018 may need to beoptimized for lower power operation compare to its counterpart used inthe corresponding training subsystem. The detector subsystem may alsohave more stringent latency requirements compared to the trainingsystem. If this is the case, gesture envelope detector logic 1018 usedin the detector subsystem may need to be designed and implemented forlower latency compared to its counterpart used in the correspondingtraining subsystem.

An output of feature generator logic 1020 is fed into feature generatorlogic 1020, which classifies the gesture based on the trained classifiermodule from its corresponding training subsystem. The ClassificationOutput may include one or more labels. Optionally, Detector 1022 mayalso assign a confidence level to each label.

Classification on Combination of Temporal and Non-Temporal Data Inputs

In another embodiment, inputs into the classification system may includea combination of temporal and non-temporal data. FIG. 11 is anillustrative example of a training subsystem in accordance with at leastone embodiment of the present disclosure where at least some of the datainputs are temporal and at least some of the data inputs arenon-temporal. Other implementations are also possible.

Non-temporal training data 1129 do not need to be processed by gestureenvelope detector 1125 and feature generator Logic 1127. Non-temporaltraining data 1129 may be fed directly into the label mapper logic 1132along with labels 1131. In some embodiments, non-temporal training datamay be processed by a separate feature generator module, non-temporalfeature generator module 1130, to extract specific non-temporal featuresof interest, which are then fed into Label mapper logic 1132. Labelmapper logic 1132 may assign the labels 1131, along with non-temporalfeatures 1136 that are attached to the label to gestures using methodssimilar to the methods for mapping labels to gestures that have beendescribed herein.

FIG. 12 is an illustrative example of a classification detectorsubsystem in accordance with at least one embodiment of the presentdisclosure where at least some of the data inputs are temporal and atleast some of the data inputs are non-temporal.

Unsupervised Classification of Temporal Data Inputs

In yet another embodiment of the present disclosure, deep learningalgorithms may be used for machine classification. Classification usingdeep learning algorithms is sometimes referred to as unsupervisedclassification. With unsupervised classification, the statistical deeplearning algorithms perform the classification task based on processingof the data directly, thereby eliminating the need for a featuregeneration step.

FIG. 13 shows an illustrative example of a classifier training subsystemin accordance with at least one embodiment of the present disclosurewhere the classifier training module is based on statistical deeplearning algorithms for unsupervised classification.

Gesture envelope detector 1349 computes gestures 1350 with correspondinggesture time envelopes from temporal training data 1348. Data segmentor1351 assigns the proper data segment or data segments to each gesturebased on information in the gesture time envelope. As an example, datasegmentor 1351 may look at the start and end time information in thegesture time envelope and assign one or more data segments thatcorrespond to the overall gesture duration. This is just one example.Data segments may be selected based on different segments orsub-segments defined by the gesture time envelope. Data segments couldalso be selected based on time segments that are outside of the gesturetime envelope but directly or indirectly related to the gesture timeenvelope. An example could be selection of data segments correspondingto a period of time immediately preceding the start of the gesture timeenvelope or selection of data segments corresponding to a period of timeimmediately following the end of the gesture time envelope. Otherexamples of time segments that are outside the gesture time envelope butdirectly or indirectly related to the gesture time envelope are alsopossible.

Gestures including data segments, gesture time envelope information andlabels are fed into classifier training module 1356. In some embodimentsof the present disclosure, only a subset of the gesture time envelopeinformation may be fed into classifier training module 1356. In someembodiments of the present disclosure, gesture time envelope informationmay be processed before it is being applied to classifier trainingmodule 1356. One example could be to make the time reference of thegesture time envelope align with the start of the data segment, ratherthan with the time base of the original temporal training data stream.Other examples are also possible. By adding time envelope informationthat further characterizes the data segments, the performance of theclassifier training module may be improved.

For example, in case of gesture recognition of eating gestures based onmotion sensor data inputs, feeding additional anchor time informationsuch as the time when the pitch angle, roll or yaw reaches a maximum orminimum into the classifier training module can improve the performanceof a trained classifier 1357 as trained classifier 1357 can analyze thetraining data and look for features and correlations specifically aroundsaid anchor times. Other examples of time envelope information that canbe fed into the classifier training module are also possible.

FIG. 14 shows an illustrative example of a classification detectorsubsystem in accordance with at least one embodiment of the presentdisclosure that could be used in combination with classificationtraining subsystem of FIG. 13.

Classifier Ensemble

In some embodiments, multiple parallel classification systems based ongesture envelope detection may be used. An example of a system withmultiple parallel classifiers is shown in FIG. 15. The number ofparallel classification systems may vary. Each classification system1510, 1512, 1514 has its own training and detector sub-system andperforms gesture envelope detection on a different subset of thetraining data 1502 and labels 1504 inputs to detect gestures, or may usedifferent thresholds or criteria to qualify gestures. Consequently, eachindividual gesture envelope detector will generate an independent set ofgestures each with different gesture time envelopes. The featuregenerator logic of each classification system creates features for thegestures created by its corresponding gesture envelope detector logic.The features may be different for each classification system. Theclassifier model used by each of the parallel classifiers may be thesame or different, or some may be the same and others may be different.Since the gesture time envelopes and features used for training of eachclassifier model are different, the parallel classification systems willproduce different Classification Outputs 1516, 1518, 1520.

The Classification Outputs 1516, 1518, 1520 of each classificationsystem may be fed into Classifier Combiner sub-system 1522. ClassifierCombiner sub-system 1522 may combine and weigh the ClassificationOutputs 1516, 1518, 1520 of the individual classification systems 1510,1512, 1514 to produce a single, overall Classification result, CombinedClassification Output 1524. The weighing may be static or dynamic. Forexample, in case of gesture recognition, certain classifiers may performbetter at correctly predicting the gestures of one group of people,whereas other classifiers may perform better at correctly predicting thegestures of another group of people. Classifier Combiner sub-system 1522may use different weights for different users or for differentcontextual conditions to improve the performance of the overallclassifier ensemble. The trained system can then be used to processunlabeled data 1506.

Other examples of temporal problems include but are not limited toautonomous driving, driver warning systems (that alert the driver whendangerous traffic conditions are detected), driver alertness detection,speech recognition, video classification (security camera monitoring,etc.) and weather pattern identification.

Ignoring the temporal nature of the data inputs as well as any featuresthat are linked to the temporal envelope of the data inputs can limitperformance of the classifier and make the classifier non-suitable forclassification tasks where a reliable detection depends on features thatare inherently linked to segments of the variable time envelope or tothe overall variable time envelope. Performance and usability can breakdown if a proper time period cannot be determined reliably, or where thetime period varies from gesture-to-gesture, from person-to-person etc.

As described herein, improved methods frame temporal problems with avariable time envelope, so that information tied to the overall variabletime envelope or to segments thereof can be extracted and included inthe feature set used to train the classifier. The proposed improvedmethods improve performance and reduce the amount of training dataneeded since features can be defined relative to the time bounds of thevariable time envelope, thereby reducing sensitivities to time and uservariances.

In addition to finding time envelopes for gestures, the system can alsofind event time envelopes. In such an approach, the system mightdetermine a gesture and a gesture envelope, but then do so foradditional gestures and then define an event envelope, such as the startand end of an eating event.

Context to Improve Overall Accuracy

FIG. 16 shows an example of a machine classification system thatincludes a cross-correlated analytics sub-system. Classification output1602 may be fed into cross-correlated analytics sub-system 1604.Cross-correlated analytics sub-system 1604 can make adjustments basedone or more contextual clues to improve the accuracy. In the example ofgesture recognition, an example of a contextual clue could be theproximity in time to other predicted gestures. For example, eatinggestures tend to be grouped together in time as part of an eatingactivity such as a meal or a snack. As one example, Cross-correlatedanalytics sub-system 1604 could increase the confidence level that apredicted gesture is an eating gesture based on the confidence level anddegree of proximity of nearby predictions.

In another embodiment, cross-correlated analytics sub-system 1604 maytake individual predicted gestures 1614 from classification output 1602as inputs and may cluster individual predicted gestures into predictedactivities 1608. For example, cross-correlated analytics sub-system 1604may map multiple bite gestures to an eating activity such as a snack ora meal. Likewise, cross-correlated analytics sub-system 1604 could mapmultiple sip gestures to a drinking activity. Other examples of activityprediction based on gesture clustering are also possible.Cross-correlated analytics sub-system 1604 may modify the confidencelevel of a predicted gesture based on the temporal spacing and sequenceof predicted activities. As an example, Cross-correlated analyticssub-system 1604 could decrease the confidence level that a predictedgesture is an eating gesture if it is detected shortly following or amida “brushing teeth” activity. In another example, Cross-correlatedanalytics sub-system 1604 could decrease the confidence level that apredicted gesture is a drinking gesture if it is detected during orshortly after a brushing teeth activity. In this case, Cross-correlatedanalytics sub-system 1604 could decide to increase the confidence levelthat the gesture is a rinsing gesture.

Cross-correlated analytics sub-system 1604 can adjust a classificationoutput of a predicted gesture based on historical information 1612 orother non-gesture meta-data 1610 information such as location, date andtime, other biometric inputs, calendar or phone call activityinformation. For example, Cross-correlated analytics sub-system 1604 mayincrease the confidence level that a predicted gesture is an eatinggesture or a predicted activity is an eating activity if GPS coordinatesindicate that the person is at a restaurant. In another example,Cross-correlated analytics sub-system 1604 may increase the confidencelevel that a predicted gesture is an eating gesture or a predictedactivity is an eating activity if it occurs at a time of day for whichpast behavior indicates that the user typically engages in eating atthis time of the day. In yet another example of the present disclosure,cross-correlated analytics sub-system 1604 may increase the confidencelevel that a predicted gesture is an eating gesture or that a predictedactivity is an eating activity if the predicted gesture or predictedactivity is preceding or following a calendar event or phone callconversation if past behavior indicates that the user typically eatspreceding or following similar calendar events (e.g., with sameattendee(s), at certain location, with certain meeting agenda, etc.) orphone call conversation (e.g., from specific phone number). While theabove examples reference eating, it will be apparent to one skilled inthe art that this could also be applied to gestures other than eating.In the general case, the machine classifier with cross-correlatedanalytics sub-system uses contextual clues, historical information andinsights from proximity sensing in time to improve accuracy, where thespecific contextual clues, historical information and insights fromproximity sensing in time and how they are applied is determined bymethods disclosed or suggested herein.

In some embodiments of the present disclosure, Classification Output1602 may include additional features or gesture time envelopeinformation. Cross-correlated analytics sub-system 1604 may process suchadditional features or gesture time envelope information to determine orextract additional characteristics of the gesture or activity. As anexample, in one embodiment of the present disclosure, Cross-correlatedanalytics sub-system 1604 derives the estimated duration of the drinkinggesture from the gesture time envelope and this information can be usedby cross-correlated analytics sub-system 1604 or by one or more systemsthat are external to the machine classifier system to estimate the fluidintake associated with the drinking gesture.

In another embodiment, Cross-correlated analytics sub-system 1604 mayderive the estimated duration of an eating gesture from the gesture timeenvelope and this information may be used by the cross-correlatedanalytics sub-system 1604 or by one or more systems that are external tothe machine classifier system to estimate the size of the biteassociated with the eating gesture. Cross-correlated analyticssub-system 1604 may combine the predicted drinking gestures with othersensor data to predict more accurately if someone is consuming a drinkthat contains alcohol and estimate the amount of alcohol consumed.Examples of other sensor data may include but are not limited tomeasuring hand vibration, heart rate, voice analysis, skin temperature,measuring blood, breath chemistry or body chemistry.

Detector sub-system 1600 may predict a specific eating or drinkingmethod and cross-correlated analytics sub-system 1604 may combine theinformation obtained from detector sub-system 1600 about specifics ofthe eating or drinking method with additional meta-data to estimate thecontent, the healthiness or the caloric intake of the food. Examples ofeating/drinking methods may include but are not limited to eating withfork, eating with knife, eating with spoon, eating with fingers,drinking from glass, drinking from cup, drinking from straw, etc.).Examples of meta-data may include but are not limited to time of day,location, environmental or social factors.

Interpretation

Conjunctive language, such as phrases of the form “at least one of A, B,and C,” or “at least one of A, B and C,” unless specifically statedotherwise or otherwise clearly contradicted by context, is otherwiseunderstood with the context as used in general to present that an item,term, etc., may be either A or B or C, or any nonempty subset of the setof A and B and C. For instance, in the illustrative example of a sethaving three members, the conjunctive phrases “at least one of A, B, andC” and “at least one of A, B and C” refer to any of the following sets:{A}, {B}, {C}, {A, B}, {A, C}, {B, C}, {A, B, C}. Thus, such conjunctivelanguage is not generally intended to imply that certain embodimentsrequire at least one of A, at least one of B and at least one of C eachto be present.

Operations of processes described herein can be performed in anysuitable order unless otherwise indicated herein or otherwise clearlycontradicted by context. Processes described herein (or variationsand/or combinations thereof) may be performed under the control of oneor more computer systems configured with executable instructions and maybe implemented as code (e.g., executable instructions, one or morecomputer programs or one or more applications) executing collectively onone or more processors, by hardware or combinations thereof. The codemay be stored on a computer-readable storage medium, for example, in theform of a computer program comprising a plurality of instructionsexecutable by one or more processors. The computer-readable storagemedium may be non-transitory.

The use of any and all examples, or exemplary language (e.g., “such as”)provided herein, is intended merely to better illuminate embodiments ofthe invention and does not pose a limitation on the scope of theinvention unless otherwise claimed. No language in the specificationshould be construed as indicating any non-claimed element as essentialto the practice of the invention.

Further embodiments can be envisioned to one of ordinary skill in theart after reading this disclosure. In other embodiments, combinations orsub-combinations of the above-disclosed invention can be advantageouslymade. The example arrangements of components are shown for purposes ofillustration and it should be understood that combinations, additions,re-arrangements, and the like are contemplated in alternativeembodiments of the present invention. Thus, while the invention has beendescribed with respect to exemplary embodiments, one skilled in the artwill recognize that numerous modifications are possible.

For example, the processes described herein may be implemented usinghardware components, software components, and/or any combinationthereof. The specification and drawings are, accordingly, to be regardedin an illustrative rather than a restrictive sense. It will, however, beevident that various modifications and changes may be made thereuntowithout departing from the broader spirit and scope of the invention asset forth in the claims and that the invention is intended to cover allmodifications and equivalents within the scope of the following claims.

All references, including publications, patent applications, andpatents, cited herein are hereby incorporated by reference to the sameextent as if each reference were individually and specifically indicatedto be incorporated by reference and were set forth in its entiretyherein.

What is claimed is:
 1. A method of sensing wearer activity, including atleast one food intake event, using at least one electronic device wornby a wearer, the method comprising: determining, using a processor,sensor readings, wherein at least one sensor reading is from anaccelerometer of the at least one electronic device that measuresmovement of an arm of the wearer; determining a first gesture indicativeof a potential event from the sensor readings; determining, using theprocessor, a stored state, the stored state being a state from a set ofstates that includes an in-event state and an out-of-event state; whenin the out-of-event state, determining, using the processor and based onthe first gesture, a start of the at least one food intake event;providing storage for event-specific parameters, initialized for thefood intake event; determining, when in the in-event state, a sequenceof gestures related to the food intake event, using the processor,wherein the processor processes gesture indications differently in thein-event state and in the out-of-event state; changing, using theprocessor, the electronic device to a higher performance state inresponse to the stored state being modified from the out-of-event stateto the in-event state, wherein the higher performance state comprisesone or more of additional power supplied to sensors, additional powerbeing drawn by the sensors, a reduced latency of a communicationschannel, an increased sensor sampling rate of the sensors, additionalcomputations performed by the processor, activation of a data recorder,and/or sending a signal to an external device indicating one or more ofthe event-specific parameters; and deriving the event-specificparameters about the food intake event from the sequence of gestures,the event-specific parameters including a timestamp relative to thestart of the food intake event.
 2. The method of claim 1, furthercomprising: detecting, using the processor and based on at least some ofthe sensor readings, an end of the food intake event; and putting theelectronic device into a lower performance state.
 3. The method of claim1, wherein determining sensor readings comprises receiving signals fromone or more of the accelerometer and additional sensors for detectingthe start of the food intake event and wherein the additional sensorsinclude a gyroscope.
 4. The method of claim 1, further comprisingtraining a learning engine to predict an occurrence of a food intakeevent based on tracking of one or more of gender, age, weight,social-economic status, timing information about the food intake event,information about location of food intake event, vital signsinformation, and hydration level information.
 5. The method of claim 1,further comprising: identifying, using a learning engine, signatures ofevent time envelopes, thereby delimiting an event time envelope;identifying, using limits of the event time envelope, gestures withinthe event time envelope; and deriving, from the gestures, the foodintake event.
 6. A method of sensing wearer activity, including at leastone behavior event, using at least one electronic device worn by awearer, the method comprising: determining, using a processor, sensorreadings, wherein at least one sensor reading is from an accelerometerof the at least one electronic device that measures movement of an armof the wearer; determining a first gesture indicative of a potentialbehavior event from the sensor readings; determining a confidence levelrelated to the first gesture, wherein the confidence level relates to alevel of confidence that the first gesture was correctly detected;determining, using the processor, a stored state, the stored state beinga state from a set of states that includes an in-event state and anout-of-event state; when in the out-of-event state, determining, usingthe processor and based on the first gesture and the confidence level, astart of the at least one behavior event; providing storage forevent-specific parameters, initialized for the behavior event; when theconfidence level is below a threshold, identifying subsequent gesturesto determine the start of the behavior event, using the processor,wherein the processor processes gesture indications differently in thein-event state and in the out-of-event state; determining, when in thein-event state, a sequence of gestures related to the behavior event;changing, using the processor, the electronic device to a higherperformance state in response to the stored state being modified fromthe out-of-event state to the in-event state, wherein the higherperformance state comprises one or more of additional power supplied tosensors, additional power being drawn by the sensors, a reduced latencyof a communications channel, an increased sensor sampling rate of thesensors, additional computations performed by the processor, activationof a data recorder, and/or sending a signal to an external deviceindicating one or more of the event-specific parameters; and derivingthe event-specific parameters about the behavior event from the sequenceof gestures, the event-specific parameters including a timestamp for thestart of the behavior event.
 7. The method of claim 6, furthercomprising: detecting, using the processor and based on at least some ofthe sensor readings, an end of the behavior event; and putting theelectronic device into a lower performance state.
 8. The method of claim6, wherein determining sensor readings comprises receiving signals fromone or more of the accelerometer and additional sensors for detectingthe start of the behavior event and wherein the additional sensorsinclude a gyroscope.
 9. The method of claim 6, wherein the behaviorevent is an eating event or a drinking event, the method furthercomprising estimating one or more of a duration of the behavior event, abite count, a sip count, an eating pace, or a drinking pace.
 10. Themethod of claim 6, further comprising recording incidences of triggersthat autonomously predict a probable start of a food intake event. 11.The method of claim 10, further comprising: training a learning engineto predict an occurrence of the behavior event based on tracking of oneor more of gender, age, weight, social-economic status, timinginformation about the at least one behavior event, information aboutlocation of behavior event, and vital signs information; identifying,using the learning engine, signatures of event time envelopes, therebydelimiting an event time envelope; identifying, using limits of theevent time envelope, gestures within the event time envelope; andderiving, from the gestures, the at least one behavior event.
 12. Themethod of claim 6, further comprising: upon detection of a possiblegesture, if the confidence level is below the threshold, waiting fordetection of another gesture within a predefined time window followingthe detection of the possible gesture before determining that the startof the behavior event had occurred.
 13. An electronic system for sensinguser activity, including at least one behavior event of a user,comprising: a wearable device having an accelerometer that measuresmovement of an arm of the user when the wearable device is being worn bythe user; one or more additional sensors that, when the wearable deviceis being worn by the user, sense parameters about movement of the user;a processor capable of reading sensor readings of the parameters,wherein at least one sensor reading is from the accelerometer; memorystorage for a stored state, the stored state being a state from a set ofstates that includes an in-event state and an out-of-event state;program code executable by the processor for determining, when in theout-of-event state, based on at least some of the sensor readings, afirst gesture indicative of a potential start of the at least onebehavior event and a confidence level for the first gesture; programcode executable by the processor for determining, when in the in-eventstate, a sequence of gestures, using sensor readings, wherein theprocessor processes gesture indications differently in the in-eventstate and in the out-of-event state; program code executable by theprocessor for changing to a higher performance state in response to thestored state being modified from the out-of-event state to the in-eventstate, wherein the higher performance state comprises one or more ofadditional power supplied to sensors, additional power being drawn bythe sensors, a reduced latency of a communications channel, an increasedsensor sampling rate of the sensors, additional computations performedby the processor, activation of a data recorder, and/or sending a signalto an external device indicating one or more event-specific parameters;and program code for deriving the event-specific parameters about thebehavior event from the sequence of gestures, the event-specificparameters including a timestamp for the potential start of the behaviorevent.
 14. The electronic system of claim 13, further comprising:program code for detecting a specific gesture from the parameters basedin part on the stored state; a gyroscope sensor; and program code fordetermining, from the gyroscope sensor the accelerometer, or both, apossible gesture.
 15. The electronic system of claim 13, furthercomprising: program code for determining external metadata about ahistory of the user; and program code for using the external metadata toimprove accuracy of the confidence level.
 16. The electronic system ofclaim 13, further comprising: memory storage for a learned dataset,wherein the learned dataset comprises data usable to detect a type ofgesture based on training examples; a classifier trained on the learneddataset comprising data records related to prior gestures of the user orother users, wherein the program code for determining possible gesturesof the user takes into account outputs of the classifier, whereby theclassifier can be used in a food intake event detection system to detecta start of a food intake event, wherein the learned dataset comprisesdata related to one or of sleep pattern, stress level, dailyinteractions, and recent activities.
 17. The electronic system of claim13, further comprising: program code for determining possible gesturesof the user; program code for assigning confidence levels to thepossible gestures; program code for determining the start of a foodintake event based on a possible gesture being determined as an eatingor drinking gesture above a threshold confidence level; program code forproviding additional power to additional sensors; and program code forusing inputs of the additional sensors to improve accuracy of theconfidence level.
 18. The electronic system of claim 13, furthercomprising: program code for distinguishing eating gestures fromdrinking gestures based on accelerometer sensor data and gyroscopesensor data, wherein sensing takes into account changes in an angle ofroll of a body part of the user and/or variance of accelerometer orgyroscope readings across one or more axes for a duration of time;program code for distinguishing, based on detected rotation of a wristof the user, between an eating gesture and a drinking gesture; andprogram code for determining, from sensor data, bite counts, sip counts,cadence, and duration, to determine bite sizes and sip sizes, whereinbite sizes and sip sizes are determined based on a duration of a hand ofthe user near the mouth of the user or determined based on an amount ofrotation of the wrist of the user and wherein the sip sizes are used ina hydration tracking process.
 19. The method of claim 1, furthercomprising outputting the timestamp in a first message to a medicationmanagement system that dispenses medication to the wearer timed withfood eating events or sending a second message to the medicationmanagement system at a predetermined time relative to the start of thefood intake event.
 20. The method of claim 6, further comprisingoutputting the timestamp in a first message to the wearer timed withbehavior events or sending a second message to the wearer at apredetermined time relative to the start of the behavior event.
 21. Amethod of sensing wearer activity, including a smoking event, using atleast one electronic device worn by a wearer, the method comprising:determining, using a processor, sensor readings from a set of sensors,wherein at least one sensor reading is from an accelerometer of the atleast one electronic device that measures movement of an arm of thewearer; determining a first gesture indicative of a potential event fromthe sensor readings; determining, using the processor, a stored state,the stored state being a state from a set of states that includes anin-event state and an out-of-event state; when in the out-of-eventstate, determining, using the processor and based on the first gesture,a start of the smoking event; providing storage for event-specificparameters, initialized for the smoking event; determining, when in thein-event state, a sequence of gestures related to the smoking event,using the processor, wherein the processor processes gesture indicationsdifferently in the in-event state and in the out-of-event state; derivethe event-specific parameters about the smoking event from the sequenceof gestures, the event-specific parameters including a timestamprelative to the start of the smoking event; change, using the processor,the electronic device to a higher performance state in response to thestored state being modified from the out-of-event state to the in-eventstate, wherein the higher performance state comprises one or more ofadditional power supplied to sensors of the set of sensors, additionalpower being drawn by the sensors of the set of sensors, a reducedlatency of a communications channel, an increased sensor sampling rateof the sensors of the set of sensors, additional computations performedby the processor, activation of a data recorder, and/or sending a signalto an external device indicating one or more of the event-specificparameters; and output the timestamp in a message to the wearer or sendthe message to the wearer at a predetermined time relative to the startof the smoking event.